Electronic pen with retractable nib and force sensor

ABSTRACT

An electronic pen having a retractable nib for interacting with a surface. The pen comprises a slidably movable cartridge having a nib for contacting the surface, a retraction mechanism for retracting the cartridge, an actuator coupled to the retraction mechanism, a force sensor providing a reaction to an axial force transmitted from the cartridge, and a processor configured for generating force data indicative of a force detected by the force sensor. The retraction mechanism comprises a barrel for receiving the cartridge, which is adapted for seating the cartridge in either an extended position or a retracted position, whereby the axial force from the cartridge is transmitted to the barrel in the extended position. A biasing mechanism biases the cartridge towards the extended or retracted positions.

FIELD OF THE INVENTION

The present invention relates to electronic pens for interacting withinteractive paper. It has been developed to provide an electronic penhaving improved interactive functionality, which can be handled and usedin the same way as conventional ballpoint pens

BACKGROUND OF THE INVENTION

The Applicant has developed the Netpage system, which is discussed belowand discussed in detail in the cross-referenced documents identifiedherein. As the invention is particularly well suited to this system, itwill be described in a Netpage context. However, it will be appreciatedthat hand-held optical sensing devices have broad ranging applicationsin many different fields and the invention is not limited to its usewithin the Netpage system.

The Netpage system involves the interaction between a user and acomputer network (or stand alone computer) via a pen and paper basedinterface. The ‘pen’ is an electronic stylus with a marking ornon-marking nib and an optical sensor for reading a pattern of codeddata on the paper (or other surface).

One of the primary features of the Netpage pen is its ability to ‘click’on interactive elements on a Netpage in the same way a mouse can clickon screen-based interactive elements (e.g. hyperlinks and so on).However, with a Netpage pen, the user simply puts the nib on theinteractive element in order to click on it. The optical sensoridentifies the element via its unique page and location ID while a forcesensor registers a ‘pen down’ condition when the nib is pressed againstthe page. Registering ‘pen down’ and ‘pen up’ is also fundamental tocapturing the user's handwriting on Netpage input fields. Non-binaryforce signals are also captured for reproducing hand-drawn strokes withvarying force-related width and opacity. Force variation can also beused as one of the parameters examined during signature verification.

The high tolerances and functionality required in force-sensing andimage-sensing pens needs to be reconciled with the need for a pen, whichis robust and easy for users to handle and transport.

Electronic image-sensing pens manufactured under license from Anoto,Inc. (see U.S. Pat. No. 7,832,361) typically use a force sensitiveresistor for measuring nib force. However, force sensitive resistorshave relatively long return-to-zero response times and are unsuitablefor detecting end-strokes with a high degree of accuracy.

SUMMARY OF THE INVENTION

In a first aspect, there is provided an electronic pen having aretractable nib for interacting with a surface, the pen comprising:

a slidably movable cartridge having a nib at a first end for contactingthe surface, the cartridge being configurable in an extended position,wherein the nib protrudes from a body of the pen, and a retractedposition wherein the nib is retracted in the body;

a retraction mechanism comprising:

-   -   a barrel for receiving an opposite second end of the cartridge,        the barrel being adapted for seating the cartridge in either the        extended position or the retracted position, whereby a        longitudinal axial force from the cartridge is transmitted to        the barrel in the extended position; and    -   a biasing mechanism for biasing the cartridge towards the        extended or retracted positions;

an actuator coupled to the retraction mechanism, the actuator enabling auser to move the cartridge between the extended and retracted positions;

a force sensor coupled to the barrel, the force sensor providing areaction to the axial force transmitted to the barrel from thecartridge; and

a processor configured for generating force data indicative of a forcedetected by the force sensor.

The pen provides a convenient means for users to retract the nib usingone hand only, whilst still having a high-sensitivity force sensor whichcan detect the slightest nib forces (e.g. 1-20 g or 5-12 g) when the nibis in the extended position.

Optionally, the pen further comprises an image sensor for imaging atleast some of a coded data pattern disposed on the surface, wherein theprocessor is configured for generating interaction data using one ormore imaged portions of the coded data pattern.

Optionally, the processor is configured to determine, using the forcedata, either one of:

a pen-down state when the nib is in contact with the surface; and

a pen-up state when the nib is not in contact with the surface.

Optionally, the processor is configured to indicate a pen-down stateonly if the force exceeds a predetermined threshold force.

Optionally, the processor is configured to generate the interaction dataonly when a pen-down state has been determined

Optionally, the image sensor is configured to sense the coded data onlywhen a pen-down state has been determined.

Optionally, the actuator comprises a manually-operable button coupled tothe cartridge, wherein depression and release of the button switches thecartridge between the retracted and extended positions.

Optionally, the pen comprises a sensing arrangement for sensing aconfiguration of the pen, the sensing arrangement comprising:

-   -   a button sensor coupled to the button, the button sensor sensing        actuation of the button; and    -   a cartridge sensor cooperating with the cartridge, the cartridge        sensor sensing whether the cartridge is extended or retracted,        wherein the processor is adapted to configure a state of the pen        in response to one or more input signals from the sensing        arrangement.

Optionally, the button sensor comprises an electronic switch, the switchbeing mechanically actuable by part of the button.

Optionally, the barrel comprises a longitudinally slidable ratchet forretaining the second end of the cartridge, the ratchet comprising aplurality of radial ratchet-teeth for longitudinal sliding engagementwith complementary longitudinal grooves defined in an inner surface ofthe barrel.

Optionally, the cartridge is replaceable and the ratchet is configuredfor releasably retaining the second end of the cartridge.

Optionally, each longitudinal groove has a cammed lip configured suchthat abutting engagement of a first cammed end-surface of theratchet-teeth onto the lip causes rotation of the ratchet.

Optionally, the barrel comprises a first abutment surface correspondingto the extended position and a second abutment surface corresponding tothe retracted position, the ratchet being rotatable and slidable towardsthe first or second abutment surface.

Optionally, the button is coupled to the ratchet via a plunger, theplunger comprising a plurality of radial plunger-teeth slidingly engagedin the longitudinal grooves, each plunger-tooth having a second cammedend-surface for abutting engagement with the first cammed end-surfacesof the ratchet-teeth, the first and second cammed end surfaces beingconfigured to impart rotation to the ratchet.

Optionally, the button is biased towards a decoupled position in whichthe button is decoupled from the plunger.

Optionally, the barrel comprises an integral pin engaged with the forcesensor, wherein a longitudinal axial force transmitted through thecartridge is detected by the force sensor via the pin.

Optionally, the force sensor is a capacitive force sensor, the pin beingengaged with a resilient moving plate of the capacitive force sensor.

Optionally, the pen comprises a preload bias assembly for biasing thebarrel and thereby the pin towards engagement with the force sensor.

Optionally, the pen comprises a housing for the barrel, wherein thepreload bias assembly comprises a preload spring engaged between thehousing and the barrel.

Optionally, the cartridge is selected from any one of:

-   -   an ink cartridge comprising an ink reservoir for supplying ink        to the nib, wherein the nib is a marking nib; and    -   a stylus cartridge wherein the nib is a non-marking nib.

In a second aspect, there is provided an electronic pen for interactingwith a surface, the pen comprising:

a cartridge having a nib for contacting the surface;

a capacitive force sensor, cooperating with the cartridge, for sensing anib force transmitted axially through the cartridge; and

a processor configured to generate force data indicative of a forcedetected by the force sensor,

wherein the capacitive force sensor comprises:

a spring having an integral conductive moving plate, the springresiliently biasing the moving plate towards a reference position;

a conductive fixed plate; and

a dielectric separating the plates.

The pen according to the second aspect advantageously provides fastresponse times (e.g. 0.1-5 ms or 0.5-2 ms) and, in particular, fastreturn-to-zero response times. These fast response times are necessarywhen capturing a continuous stream of digital ink during handwritteninput on a page. The capacitive force sensor defined in the secondaspect may be incorporated into any of the electronic pens describedherein. It is particularly suitable for use in electronic pens havingretractable cartridges.

Optionally, the spring is comprised of a material selected from thegroup consisting of: stainless steel and beryllium-copper alloy.

Optionally, the nib force moves the moving plate towards the fixedplate.

Optionally, the nib force moves the moving plate away from the fixedplate.

Optionally, the spring comprises one or more spring arms supporting themoving plate.

Optionally, the one or more spring arms extend radially,circumferentially or spirally from the moving plate to an outer annularsupport.

Optionally, the moving plate and the one or more spring arms arecoplanar in the reference position.

Optionally, the moving plate and the one or more spring arms are formedfrom a single blank of material.

Optionally, the dielectric comprises air and a polymer.

Optionally, the polymer is polyimide.

Optionally, the moving plate and the fixed plate are separated by arigid spacer.

Optionally, the nib is retractable within a body of the pen.

Optionally, the cartridge is coupled to the force sensor via aretraction mechanism.

Optionally, the retraction mechanism comprises a user-operable buttonfor retracting and extending the nib.

In a third aspect, there is provided an electronic pen comprising:

a user-replaceable cartridge having a nib at a first end and an oppositesecond end;

a force sensor for sensing a nib force transmitted axially through thecartridge from the nib;

a retraction mechanism comprising:

-   -   a barrel for receiving the second end of the cartridge, the        barrel being adapted for seating the cartridge in either an        extended position or a retracted position, the barrel being        engaged with the force sensor; and    -   a first biasing means for biasing the cartridge towards the        extended or retracted positions; and

a second biasing means for biasing the barrel towards engagement withthe force sensor.

Advantageously, the use of two biasing means in the pen enablesconvenient nib retraction/extension whilst still allowing extraction andreplacement of the cartridge from the pen without damaging the forcesensor.

Optionally, the barrel comprises a boot for releasably retaining thesecond end of the cartridge.

Optionally, the boot is longitudinally slidable into either the extendedposition or the retracted position.

Optionally, the cartridge is extractable from the boot, and thereby thepen, by pulling the cartridge against the bias of the first biasingmeans.

Optionally, the first biasing means comprises a spring engaged betweenthe barrel and the boot.

Optionally, the barrel comprises an end-stop abutment surface and theboot is configured to release the cartridge upon engagement with theend-stop abutment surface.

Optionally, the second biasing means is configured to maintain thebarrel in engagement with the force sensor when the boot is engaged withthe end-stop abutment surface of the barrel.

Optionally, the boot comprises a retaining sheath for retaining thesecond end of the cartridge.

Optionally, the pen further comprises a manually-operable button coupledto the retraction mechanism.

Optionally, the pen further comprises a housing for the barrel, whereinthe second biasing means comprises a spring engaged between the barreland the housing.

In a fourth aspect, there is provided an electronic pen for interactingwith a surface, the pen comprising:

a retractable cartridge having a nib;

an automatic retraction mechanism for automatically retracting thecartridge such that the nib is retracted within a body of the pen; and

a first processor for controlling the automatic retraction mechanism,

wherein the first processor is configured to automatically retract thecartridge in response to one or more predetermined pen conditions.

The pen according to the fourth aspect advantageously minimizes damageto the pen by retracting the nib under a set of predetermined penconditions. For example, accelerometers may detect when the pen is infreefall, and signal to the pen to automatically retract the nib beforeimpact in order to minimize damage.

Optionally, the pen comprises one or more input sensors for sensing theone or more pen conditions, the one or more input sensors providing oneor more input signals to the first processor.

Optionally, the one or more input sensors are selected from the groupconsisting of:

-   -   a force sensor coupled to the cartridge, the force sensor        sensing a nib force transmitted axially through the cartridge        from the nib;    -   a nib switch coupled to the cartridge, the nib switch being        actuated when a predetermined force is exerted on the nib; and    -   an image sensor for imaging at least some of a coded data        pattern disposed on the surface, wherein the first processor        and/or a second processor is configured for generating        interaction data using one or more imaged portions of the coded        data pattern.

Optionally, the first processor is configured to automatically retractthe cartridge if the first processor determines any one of:

the nib force is greater than a predetermined threshold force

the surface is not coded with the coded data pattern;

a quality of the coded data pattern falls below a predeterminedthreshold;

a quality of the interaction data falls below a predetermined threshold;

a buffer for storing the interaction data is full;

a buffer for storing the interaction data has available memory which islower than a predetermined threshold;

the pen is moved relative to the surface with a speed exceeding apredetermined threshold; and

the pen is tilted relative to the surface with a tilt exceeding apredetermined threshold.

Optionally, the one or more input sensors comprises at least one sensorselected from the group consisting of: an accelerometer, a timer, anenvironmental temperature sensor, an environmental humidity sensor, abattery status sensor and a connectivity sensor.

Optionally, the first processor is configured to automatically retractthe cartridge if the first processor determines any one of:

the pen is in freefall motion;

the pen experiences an impact shock;

a period of inactivity exceeds a predetermined threshold;

an environmental temperature exceeds a predetermined threshold;

an environmental humidity exceeds a predetermined threshold.

a battery capacity is lower than a predetermined threshold;

a connection to a remote computer system or network is broken.

Optionally, retraction of the cartridge configures the pen in apower-down state.

Optionally, a user-operable switch is coupled to the automaticretraction mechanism and/or the first processor.

Optionally, the pen comprises an extension mechanism for extending thecartridge, such that the nib protrudes from the body of the pen.

Optionally, extension of the cartridge configures the pen in a power-upstate whereby the force sensor is responsive to detecting the nib force,the image sensor is responsive to imaging the coded data pattern and/orthe processor is responsive to generating interaction data.

Optionally, the extension mechanism is disabled if the first processordetermines that a nib disablement condition exists.

Optionally, the extension mechanism is an automatic extension mechanismfor automatically extending the cartridge.

Optionally, the automatic retraction mechanism comprises the automaticextension mechanism.

Optionally, the retraction/extension mechanism is coupled to auser-operable switch, and wherein the effect of the switch depends onwhether the cartridge is extended or retracted.

Optionally, the pen comprises a manual extension mechanism.

Optionally, the pen further comprises a manual retraction mechanism.

Optionally, the automatic retraction mechanism comprises a motor coupledto a cartridge retraction mechanism.

Optionally, the automatic retraction mechanism comprises a motor coupledto a force sensor assembly, wherein the force sensor assembly is coupledto the cartridge.

Optionally, the automatic retraction mechanism comprises an actuatorconfigured for releasing a cartridge assembly from a catch, thecartridge assembly comprising the cartridge, wherein the cartridgeassembly is biased towards a retracted position.

In a fifth aspect, there is provided an electronic pen comprising:

-   -   a retractable cartridge having a nib at a first end for        contacting a surface, the cartridge being configurable in an        extended position, wherein the nib protrudes from a body of the        pen, and a retracted position wherein the nib is retracted in        the body;    -   a retraction mechanism for extending and retracting the        cartridge;    -   a button for actuating the retraction mechanism;    -   a sensing arrangement for sensing a configuration of the pen,        the sensing arrangement comprising:        -   a button sensor coupled to the button, the button sensor            sensing actuation of the button; and        -   a cartridge sensor cooperating with the cartridge, the            cartridge sensor sensing whether the cartridge is extended            or retracted; and    -   a processor cooperating with the sensing arrangement, the        processor being adapted to configure a state of the pen in        response to one or more input signals from the sensing        arrangement.

The pen according to the fifth aspect advantageously saves battery powerby powering down the pen when it is detected that the pen is not in use.The button sensor and the cartridge sensor work in conjunction toprovide the requisite input signals to the processor.

Optionally, the processor is adapted to configure the pen in either anactive state or a quiescent state in response to the one or more inputsignals from the sensing arrangement.

Optionally, the button sensor comprises an electronic switch, the switchbeing mechanically actuable by part of the button.

Optionally, the cartridge sensor comprises an optical sensor.

Optionally, the optical sensor is positioned for sensing at least partof the cartridge.

Optionally, the cartridge comprises first and second opticallydetectable regions, the first and second regions being distinguishableby the optical sensor so that sensing of the first region indicates anextended position and sensing of the second region indicates a retractedposition

Optionally, either one of the first or second regions is marked so as tocontrast optically with a body of the cartridge.

Optionally, the pen comprises an image sensor for imaging at least someof a coded data pattern disposed on the surface, wherein the processoris configured for generating interaction data using one or more imagedportions of the coded data pattern.

Optionally, the image sensor is configured to sense the coded datapattern only if the processor has determined that the nib is in theextended position.

Optionally, the sensing arrangement further comprises a force sensor forsensing a nib force transmitted axially through the cartridge from thenib, wherein the processor is configured to generate force dataindicative of the nib force.

Optionally, the processor is configured to determine, using the forcedata, either one of:

a pen-down state when the nib is in contact with the surface; and

a pen-up state when the nib is not in contact with the surface.

Optionally, the image sensor is configured to sense the coded datapattern only if the processor has determined a pen-down state.

Optionally, the processor is configured to generate the interaction dataonly if the processor has determined a pen-down state.

Optionally, the sensing arrangement further comprises a timer.

Optionally, the processor is adapted to configure the pen in a quiescentstate if the timer indicates that a predetermined threshold period ofinactivity has been exceeded.

Optionally, the retraction mechanism comprises:

-   -   a barrel for receiving a second end of the cartridge opposite        the first end, the barrel being adapted for seating the        cartridge in either the extended position or the retracted        position; and    -   a biasing mechanism for biasing the cartridge towards the        extended or retracted positions.

Optionally, the barrel comprises a boot for releasably retaining thesecond end of the cartridge.

Optionally, the boot comprises a retaining sheath for retaining thesecond end of the cartridge.

Optionally, the boot is longitudinally slidable into either the extendedposition or the retracted position.

In a sixth aspect, there is provided an electronic pen comprising:

-   -   a retractable cartridge having a nib at a first end for        contacting a surface, the cartridge being configurable in an        extended position, wherein the nib protrudes from a body of the        pen, and a retracted position wherein the nib is retracted in        the body;    -   a force-actuable device, the device being actuated by a nib        force transmitted axially through the cartridge from the nib;    -   a retraction mechanism comprising a barrel for receiving an        opposite second end of the cartridge, the barrel being adapted        for seating the cartridge in either an extended position or a        retracted position, the barrel being engaged with the        force-actuable device;    -   a button for actuating the retraction mechanism; and    -   a decoupling mechanism for biasing the button away from coupled        engagement with the retraction mechanism and thereby the        force-actuable device.

In the pen according to the sixth aspect, the accuracy of theforce-actuable device (e.g. force sensor) is advantageously improved bybiasing the actuation button away from the retraction mechanism andthereby the force-actuable device engaged with the retraction mechanism.

Optionally, the force-actuable device is selected from the groupconsisting of:

a force sensor coupled to the cartridge for sensing a nib forcetransmitted axially through the cartridge from the nib; and

a nib switch coupled to the cartridge, the nib switch being actuatedwhen a predetermined force is exerted on the nib.

Optionally, the pen comprises a processor configured for generatingforce data indicative of a force sensed by the force sensor.

Optionally, manual actuation of the button couples the button to theretraction mechanism against a bias of the decoupling mechanism.

Optionally, the decoupling mechanism comprises a decoupling springengaged between the button and a support surface.

Optionally, the button is couplable to the retraction mechanism viaabutting engagement with a slidably movable plunger, the plunger passingthrough a grommet supported by the pen body, wherein the grommet definesthe support surface.

Optionally, the retraction mechanism comprises:

-   -   a barrel comprising a first abutment surface corresponding to        the extended position and a second abutment surface        corresponding to the retracted position; and    -   a ratchet slidably mounted in the barrel for retaining the        second end of the cartridge,        wherein the plunger abuts with the ratchet such that actuation        and release of the button causes sliding and rotating movement        of the ratchet towards the first or second abutment surface.

Optionally, a first biasing means biases the ratchet towards the firstor second abutment surface against the decoupling bias of the decouplingbiasing means.

Optionally, the ratchet comprises a retaining sheath for releasablyretaining the second end of the cartridge.

Optionally, the cartridge is extractable from the retaining sheath, andthereby the pen, by pulling the cartridge against the bias of the firstbiasing means.

Optionally, comprises an end-stop abutment surface and the retainingsheath is configured to release the cartridge upon engagement with theend-stop abutment surface.

Optionally, the pen further comprises a second biasing means for biasingthe barrel towards engagement with the force sensor.

Optionally, the second biasing means is configured to maintain thebarrel in engagement with the force sensor.

Optionally, the pen further comprises a housing for the barrel, whereinthe second biasing means comprises a preload spring engaged between thebarrel and the housing.

Optionally, the pen comprises a sensing arrangement for sensing aconfiguration of the pen, the sensing arrangement comprising:

-   -   a button sensor coupled to the button, the button sensor sensing        actuation of the button; and    -   a cartridge sensor cooperating with the cartridge, the cartridge        sensor sensing whether the cartridge is extended or retracted,        wherein the processor is adapted to configure a state of the pen        in response to one or more input signals from the sensing        arrangement.

Optionally, the force sensor described herein in respect of the first,third, fourth, fifth and sixth aspects is a capacitive force sensor asdefined in respect of the second aspect (and any optional embodimentsthereof).

As used herein, the term “cartridge” is used to refer to either an inkcartridge or a stylus cartridge. An ink cartridge has a marking nib atone end and contains an ink reservoir for supplying ink to the markingnib. A stylus cartridge has a non-marking nib at one end and does notcontain an ink reservoir, the stylus cartridge being in employed inpointing instruments rather than writing instruments. Either type ofcartridge is typically an elongate shaft having a longitudinal axis anda nib at one end. The cartridge is usually right-cylindrical, althoughother forms are of course possible and within the ambit of the presentinvention. Ink cartridges and stylus cartridges are typicallyinterchangeably replaceable in the electronic pens (e.g. Netpage pens)described herein. Hence a stylus pen comprising a stylus cartridge maybe converted into a marking pen comprising an ink cartridge and viceversa.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described by way of exampleonly with reference to the accompanying drawings in which:

FIG. 1 is a perspective view of a Netpage pen;

FIG. 2 is a perspective view of a nib end of the Netpage pen;

FIG. 3 is a schematic of a the relationship between a sample printednetpage and its online page description;

FIG. 4 shows an embodiment of basic netpage architecture with variousalternatives for the relay device;

FIG. 5 shows the structure of a tag;

FIG. 6 shows a group of twelve data symbols and four targets;

FIG. 7 shows the layout of a 2-6PPM and 3-6PPM data symbol;

FIG. 8 shows the spacing of macrodot positions;

FIG. 9 shows the layout of a 2-6PPM registration symbol;

FIG. 10 shows a semi-replicated x-coordinate codeword X;

FIG. 11 shows a semi-replicated y-coordinate codeword Y;

FIG. 12 shows common codewords A, B, C and D, with codeword A shown inbold outline;

FIG. 13 shows an optional codeword E;

FIG. 14 shows the layout of a complete tag;

FIG. 15 shows the layout of a Reed-Solomon codeword;

FIG. 16 is a flowchart of image processing;

FIG. 17 is a cross-sectional front view of the Netpage pen;

FIG. 18 is a perspective view showing cradle contacts on the Netpagepen;

FIG. 19 is a perspective view of the Netpage pen docked in a Netpagecradle;

FIGS. 20A-D show schematically various charging and data connectionoptions for the Netpage pen and Netpage cradle;

FIG. 21 is an exploded view of the pen;

FIG. 22 is a longitudinal section of the pen;

FIG. 23 is an exploded view of an optical assembly for the pen;

FIG. 24 is a cutaway perspective of the optical assembly;

FIG. 25 is a perspective view of a force sensing assembly for the pen;

FIG. 26 is an exploded view of a force sensor;

FIG. 27 shows a force sensor spring;

FIG. 28 shows schematically a PCB stack comprising the force sensor;

FIG. 29 shows schematically a PCB stack for an alternative invertedforce sensor;

FIG. 30 is a perspective view of a ballpoint pen cartridge;

FIG. 31 is an exploded view of a nib retraction assembly for the pen;

FIG. 32A is a cutaway view of the barrel showing ratchet and plunger inan extended position;

FIG. 32B shows the barrel of FIG. 32A with the plunger removed forclarity;

FIG. 33 shows schematically a first automatic retraction mechanismcomprising a ratchet mechanism;

FIG. 34 shows schematically a second automatic retraction mechanism withdirect coupling to a force sensor assembly;

FIG. 35 shows schematically a third automatic retraction mechanismincorporating a manual extension mechanism;

FIG. 36 is an interconnect diagram for a main PCB of the pen;

FIGS. 37A and 37B are upper and lower perspective views of the main PCB;

FIG. 38 is a perspective view of a cradle contacts leadframe insert forthe pen;

FIG. 39 is a perspective view of a gesture flex PCB for the pen;

FIGS. 40A and 40B are longitudinal sections through pen optics;

FIG. 41 is a ray trace for the pen optics alongside the pen cartridge;and

FIG. 42 is block diagram of the pen electronics.

DETAILED DESCRIPTION

As discussed above, the invention is well suited for incorporation inthe Assignee's Netpage system. In light of this, the invention has beendescribed as a component of a broader Netpage architecture. However, itwill be readily appreciated that electronic pens have much broaderapplication in many different fields. Accordingly, the present inventionis not necessarily restricted to a Netpage context.

1.1 Netpage System Architecture

In a preferred embodiment, the invention is configured to work with thenetpage networked computer system, a detailed overview of which follows.It will be appreciated that not every implementation will necessarilyembody all or even most of the specific details and extensions discussedbelow in relation to the basic system. However, the system is describedin its most complete form to reduce the need for external reference whenattempting to understand the context in which the preferred embodimentsand aspects of the present invention operate.

In brief summary, the preferred form of the netpage system employs acomputer interface in the form of a mapped surface, that is, a physicalsurface which contains references to a map of the surface maintained ina computer system. The map references can be queried by an appropriatesensing device. Depending upon the specific implementation, the mapreferences may be encoded visibly or invisibly, and defined in such away that a local query on the mapped surface yields an unambiguous mapreference both within the map and among different maps. The computersystem can contain information about features on the mapped surface, andsuch information can be retrieved based on map references supplied by asensing device used with the mapped surface. The information thusretrieved can take the form of actions which are initiated by thecomputer system on behalf of the operator in response to the operator'sinteraction with the surface features.

In its preferred form, the netpage system relies on the production of,and human interaction with, netpages. These are pages of text, graphicsand images printed on ordinary paper, but which work like interactivewebpages. A surface coding pattern is printed on each page using inkwhich is substantially invisible to the unaided human eye. The ink,however, and thereby the coding pattern, can be sensed by an opticallyimaging sensing device and transmitted to the netpage system.

In the present invention, the optically imaging sensing device takes theform of a pen 400, as shown in FIG. 1. Referring to FIG. 2, the pen 400has a retractable nib 406 and an imaging system comprising a pair ofnear-infrared illumination LEDs 416 for illuminating the surface codingpattern and an optics system 412 for acquiring an image of the surfacecoding pattern. The pen 400 will be described in more detail in Section3 below.

In one embodiment of the Netpage system, active buttons and hyperlinkson each page can be clicked with the pen 400 to request information fromthe network or to signal preferences to a network server. In anotherembodiment, text written by hand on a netpage is automaticallyrecognized and converted to computer text in the netpage system,allowing forms to be filled in. In other embodiments, signaturesrecorded on a netpage are automatically verified, allowing e-commercetransactions to be securely authorized. In other embodiments, text on anetpage may be clicked or gestured to initiate a search based onkeywords indicated by the user. The netpage pen may incorporate agesture button to indicate a gesture interaction with a netpage surface.Gesture-based interactions are described in more detail in US2007/0143715, the contents of which is herein incorporated by reference.

As illustrated in FIG. 3, a printed netpage 1 can represent ainteractive form which can be filled in by the user both physically, onthe printed page, and “electronically”, via communication between thepen and the netpage system. The example shows a “Request” formcontaining name and address fields and a submit button. The netpage 1consists of graphic data 2, printed using visible ink, and a surfacecoding pattern 3 superimposed with the graphic data. The surface codingpattern 3 comprises a plurality of tags 4. One such tag 4 is shown inthe shaded region of FIG. 3, although it will be appreciated thatcontiguous tags 4, defined by the coding pattern 3, are densely tiledover the whole netpage 1.

A corresponding page description 5, stored on the netpage network,describes the individual elements of the netpage. In particular itdescribes the type and spatial extent (zone) of each interactive element(i.e. text field or button in the example), to allow the netpage systemto correctly interpret input via the netpage. The submit button 6, forexample, has a zone 7 which corresponds to the spatial extent of thecorresponding graphic 8.

As illustrated in FIG. 4, a netpage sensing device, such as the pen 400described in Section 3, usually works in conjunction with a netpagerelay device 601, which is an Internet-connected device for home, officeor mobile use. The pen 400 is wireless and communicates securely withthe netpage relay device 601 via a short-range radio link 9. In analternative embodiment, the netpage pen 400 utilises a wired connection,such as a USB or other serial connection, to the relay device 601. In afurther alternative embodiment, the netpage pen 400 may have an onboardcomputer system for interpreting interaction data without reference to aremote Netpage server 10.

The relay device 601 performs the basic function of relaying interactiondata to a page server 10, which interprets the interaction data. Asshown in FIG. 4, the relay device 601 may, for example, take the form ofa personal computer 601 a, a netpage printer 601 b or some other relay601 c (e.g. personal computer or mobile phone incorporating a webbrowser).

The netpage printer 601 b is able to deliver, periodically or on demand,personalized newspapers, magazines, catalogs, brochures and otherpublications, all printed at high quality as interactive netpages.Unlike a personal computer, the netpage printer is an appliance whichcan be, for example, wall-mounted adjacent to an area where the morningnews is first consumed, such as in a user's kitchen, near a breakfasttable, or near the household's point of departure for the day. It alsocomes in tabletop, desktop, portable and miniature versions. Netpagesprinted on-demand at their point of consumption combine the ease-of-useof paper with the timeliness and interactivity of an interactive medium.

Alternatively, the netpage relay device 601 may be a portable device,such as a mobile phone or PDA, a laptop or desktop computer, or aninformation appliance connected to a shared display, such as a TV. Ifthe relay device 601 is not a netpage printer 601 b which printsnetpages digitally and on demand, the netpages may be printed bytraditional analog printing presses, using such techniques as offsetlithography, flexography, screen printing, relief printing androtogravure, as well as by digital printing presses, using techniquessuch as drop-on-demand inkjet, continuous inkjet, dye transfer, andlaser printing.

As shown in FIG. 4, the netpage sensing device 400 interacts with aportion of the tag pattern on a printed netpage 1, or other printedsubstrate such as a label of a product item 251, and communicates, via ashort-range radio link 9, the interaction to the relay device 601. Therelay 601 sends corresponding interaction data to the relevant netpagepage server 10 for interpretation. Raw data received from the sensingdevice 400 may be relayed directly to the page server 10 as interactiondata. Alternatively, the interaction data may be encoded in the form ofan interaction URI and transmitted to the page server 10 via a user'sweb browser 601 c. The web browser 601 c may then receive a URI from thepage server 10 and access a webpage via a webserver 201. In somecircumstances, the page server 10 may access application computersoftware running on a netpage application server 13.

The netpage relay device 601 can be configured to support any number ofsensing devices, and a sensing device can work with any number ofnetpage relays. In the preferred implementation, each netpage sensingdevice 400 has a unique identifier. This allows each user to maintain adistinct profile with respect to a netpage page server 10 or applicationserver 13.

Digital, on-demand delivery of netpages 1 may be performed by thenetpage printer 601 b, which exploits the growing availability ofbroadband Internet access. Netpage publication servers 14 on the netpagenetwork are configured to deliver print-quality publications to netpageprinters. Periodical publications are delivered automatically tosubscribing netpage printers via pointcasting and multicasting Internetprotocols. Personalized publications are filtered and formattedaccording to individual user profiles.

A netpage pen may be registered with a netpage registration server 11and linked to one or more payment card accounts. This allows e-commercepayments to be securely authorized using the netpage pen. The netpageregistration server compares the signature captured by the netpage penwith a previously registered signature, allowing it to authenticate theuser's identity to an e-commerce server. Other biometrics can also beused to verify identity. One version of the netpage pen includesfingerprint scanning, verified in a similar way by the netpageregistration server.

1.2 Netpages

Netpages are the foundation on which a netpage network is built. Theyprovide a paper-based user interface to published information andinteractive services.

As shown in FIG. 3, a netpage consists of a printed page (or othersurface region) invisibly tagged with references to an onlinedescription 5 of the page. The online page description 5 is maintainedpersistently by the netpage page server 10. The page descriptiondescribes the visible layout and content of the page, including text,graphics and images. It also describes the input elements on the page,including buttons, hyperlinks, and input fields. A netpage allowsmarkings made with a netpage pen on its surface to be simultaneouslycaptured and processed by the netpage system.

Multiple netpages (for example, those printed by analog printingpresses) can share the same page description. However, to allow inputthrough otherwise identical pages to be distinguished, each netpage maybe assigned a unique page identifier. This page ID has sufficientprecision to distinguish between a very large number of netpages.

Each reference to the page description 5 is repeatedly encoded in thenetpage pattern. Each tag (and/or a collection of contiguous tags)identifies the unique page on which it appears, and thereby indirectlyidentifies the page description 5. Each tag also identifies its ownposition on the page. Characteristics of the tags are described in moredetail below.

Tags are typically printed in infrared-absorptive ink on any substratewhich is infrared-reflective, such as ordinary paper, or in infraredfluorescing ink. Near-infrared wavelengths are invisible to the humaneye but are easily sensed by a solid-state image sensor with anappropriate filter.

A tag is sensed by a 2D area image sensor in the netpage sensing device,and the tag data is transmitted to the netpage system via the nearestnetpage relay device 601. The pen 400 is wireless and communicates withthe netpage relay device 601 via a short-range radio link. It isimportant that the pen recognize the page ID and position on everyinteraction with the page, since the interaction is stateless. Tags areerror-correctably encoded to make them partially tolerant to surfacedamage.

The netpage page server 10 maintains a unique page instance for eachunique printed netpage, allowing it to maintain a distinct set ofuser-supplied values for input fields in the page description 5 for eachprinted netpage 1.

2 Netpage Tags 2.1 Tag Data Content

Each tag 4 identifies an absolute location of that tag within a regionof a substrate.

Each interaction with a netpage should also provide a region identitytogether with the tag location. In a preferred embodiment, the region towhich a tag refers coincides with an entire page, and the region ID istherefore synonymous with the page ID of the page on which the tagappears. In other embodiments, the region to which a tag refers can bean arbitrary subregion of a page or other surface. For example, it cancoincide with the zone of an interactive element, in which case theregion ID can directly identify the interactive element.

As described in the Applicant's previous applications (e.g. U.S. Pat.No. 6,832,717), the region identity may be encoded discretely in eachtag 4. As will be described in more detail below, the region identitymay be encoded by a plurality of contiguous tags in such a way thatevery interaction with the substrate still identifies the regionidentity, even if a whole tag is not in the field of view of the sensingdevice.

Each tag 4 should preferably identify an orientation of the tag relativeto the substrate on which the tag is printed. Orientation data read froma tag enables the rotation (yaw) of the pen 101 relative to thesubstrate to be determined

A tag 4 may also encode one or more flags which relate to the region asa whole or to an individual tag. One or more flag bits may, for example,signal a sensing device to provide feedback indicative of a functionassociated with the immediate area of the tag, without the sensingdevice having to refer to a description of the region. A netpage penmay, for example, illuminate an “active area” LED when in the zone of ahyperlink.

A tag 4 may also encode a digital signature or a fragment thereof. Tagsencoding (partial) digital signatures are useful in applications whereit is required to verify a product's authenticity. Such applications aredescribed in, for example, US Publication No. 2007/0108285, the contentsof which is herein incorporated by reference. The digital signature maybe encoded in such a way that it can be retrieved from every interactionwith the substrate. Alternatively, the digital signature may be encodedin such a way that it can be assembled from a random or partial scan ofthe substrate.

It will, of course, be appreciated that other types of information (e.g.tag size etc) may also be encoded into each tag or a plurality of tags,as will be explained in more detail below.

2.2 General Tag Structure

As described above in connection with FIG. 3, the netpage surface codinggenerally consists of a dense planar tiling of tags. In the presentinvention, each tag 4 is represented by a coding pattern which containstwo kinds of elements. Referring to FIGS. 5 and 6, the first kind ofelement is a target element. Target elements in the form of target dots301 allow a tag 4 to be located in an image of a coded surface, andallow the perspective distortion of the tag to be inferred. The secondkind of element is a data element in the form of a macrodot 302 (seeFIG. 8). The macrodots 302 encode data values. As described in theApplicant's earlier disclosures (e.g. U.S. Pat. No. 6,832,717), thepresence or absence of a macrodot was be used to represent a binary bit.However, the tag structure of the present invention encodes a data valueusing multi-pulse position modulation, which is described in more detailin Section 2.3.

The coding pattern 3 is represented on the surface in such a way as toallow it to be acquired by an optical imaging system, and in particularby an optical system with a narrowband response in the near-infrared.The pattern 3 is typically printed onto the surface using a narrowbandnear-infrared ink.

FIG. 5 shows the structure of a complete tag 4 with target elements 301shown. The tag 4 is square and contains sixteen target elements. Thosetarget elements 301 located at the edges and corners of the tag (twelvein total) are shared by adjacent tags and define the perimeter of thetag. In contrast with the Applicant's previous tag designs, the highnumber of target elements 301 advantageously facilitates accuratedetermination of a perspective distortion of the tag 4 when it is imagedby the sensing device 101. This improves the accuracy of tag sensingand, ultimately, position determination.

The tag 4 consists of a square array of nine symbol groups 303. Symbolgroups 303 are demarcated by the target elements 301 so that each symbolgroup is contained within a square defined by four target elements.Adjacent symbol groups 303 are contiguous and share targets.

Since the target elements 301 are all identical, they do not demarcateone tag from its adjacent tags. Viewed purely at the level of targetelements, only symbol groups 303, which define cells of a target grid,can be distinguished—the tags 4 themselves are indistinguishable byviewing only the target elements. Hence, tags 4 must be aligned with thetarget grid as part of tag decoding.

The tag 4 is designed to allow all tag data, with the exception of anembedded data object (see Section 2.9.3), to be recovered from animaging field of view substantially the size of the tag. This impliesthat any data unique to the tag 4 must appear four times within thetag—i.e. once in each quadrant or quarter; any data unique to a columnor row of tags must appear twice within the tag—i.e. once in eachhorizontal half or vertical half of the tag respectively; and any datacommon to a set of tags needs to appear once within the tag.

2.3 Symbol Groups

As shown in FIG. 6, each of the nine symbol groups 303 comprises twelvedata symbols 304, each data symbol being part of a codeword. Inaddition, each symbol group 303 comprises a pair of registrationsymbols—a vertical registration symbol (‘VRS’) and a horizontalregistration symbol (‘HRS’). These allow the orientation and/ortranslation of the tag in the field of view to be determined.Translation refers to the translation of tag(s) relative to the symbolgroups 303 in the field of view. In other words, the registrationsymbols enable alignment of the ‘invisible’ tags with the target grid.

Each data symbol 304 is a multi-pulse position modulated (PPM) datasymbol. Typically, each PPM data symbol 304 encodes a single 4-bitReed-Solomon symbol using either 2 or 3 macrodots in any of 6 positions{p₀, p₁, p₂, p₃, p₄, p₅}, i.e. using 2-6 or 3-6 pulse-positionmodulation (PPM). However, it will be appreciated that other forms ofmulti-PPM encoding are equally possible.

3-6PPM has a range of 20 codes, or 4.3 bits, and is used forReed-Solomon redundancy symbols. 2-6PPM has a range of 15 codes, or 3.9bits, and is used for Reed-Solomon data symbols. 4-bit Reed-Solomon datasymbols are converted to base 15 prior to encoding to reduce the numberof required codes per symbol to 15.

FIG. 7 shows the layout for a 2-6PPM or 3-6PPM data symbol 304.

Table 1 defines the mapping from 2-6PPM symbol values to Reed-Solomonsymbol values.

TABLE 1 2-6 PPM to Reed-Solomon symbol mapping CorrespondingReed-Solomon 2-6 PPM symbol symbol value value (p₅-p₀) (base 15) 0000110 000101 1 000110 2 001001 3 001010 4 001100 5 010001 6 010010 7 0101008 011000 9 100001 a 100010 b 100100 c 101000 d 110000 e

Table 2 defines the mapping from 3-6PPM symbol values to Reed-Solomonsymbol values. Unused symbol values can be treated as erasures.

TABLE 2 3-6 PPM to Reed-Solomon symbol mapping CorrespondingReed-Solomon 3-6 PPM symbol symbol value value (p₅-p₀) (base 16) 000111unused 001011 unused 001101 0 001110 1 010011 2 010101 3 010110 4 0110015 011010 6 011100 7 100011 8 100101 9 100110 a 101001 b 101010 c 101100d 110001 e 110010 f 110100 unused 111000 unused

2.4 Targets and Macrodots

The spacing of macrodots 302 in both dimensions, as shown in FIG. 8, isspecified by the parameter s. It has a nominal value of 127 μm, based on8 dots printed at a pitch of 1600 dots per inch.

Only macrodots 302 are part of the representation of a symbol 304 in thepattern. The outline of a symbol 304 is shown in, for example, FIGS. 5and 6 merely to elucidate more clearly the structure of a tag 4.

A macrodot 302 is nominally square with a nominal size of ( 4/8)s.However, it is allowed to vary in size by 110% according to thecapabilities of the device used to produce the pattern.

A target 301 is nominally circular with a nominal diameter of (12/8)s.However, it is allowed to vary in size by ±10% according to thecapabilities of the device used to produce the pattern.

Each symbol group 303 has a width of 10s. Therefore, each tag 4 has awidth of 30s and a length of 30s. However, it should be noted from FIG.5 that the tag 4 is configured so that some data symbols 304A extendbeyond the perimeter edge of the tag 4 by one macrodot unit (is), andinterlock with complementary symbol groups 304B from adjacent tags. Thisarrangement provides a tessellated pattern of data symbols 304 withinthe target grid. From a data acquisition standpoint, tessellation ofdata symbols in this way increases the effective length of each tag 4 byone macrodot unit.

The macrodot spacing, and therefore the overall scale of the tagpattern, is allowed to vary by 127 μm and 120 μm according to thecapabilities of the device used to produce the pattern. Any deviationfrom the nominal scale is recorded in each tag (in a tag size ID field)to allow accurate generation of position samples.

These tolerances are independent of one another. They may be refinedwith reference to particular printer characteristics.

2.5 Field of View

As mentioned above, the tag 4 is designed to allow all tag data to berecovered from an imaging field of view roughly the size of the tag. Anydata common to a set of contiguous tags only needs to appear once withineach tag, since fragments of the common data can be recovered fromadjacent tags. Any data common only to a column or row of tags mustappear twice within the tag—i.e. once in each horizontal half orvertical half of the tag respectively. And any data unique to the tagmust appear four times within the tag—i.e. once in each quadrant.

Although data which is common to a set of tags, in one or both spatialdimensions, may be decoded from fragments from adjacent tags,pulse-position modulated values are best decoded from spatially-coherentsamples, since this allows raw sample values to be compared withoutfirst being normalised. This implies that the field of view must belarge enough to contain two complete copies of each such pulse-positionmodulated value. The tag is designed so that the maximum extent of apulse-position modulated value is three macrodots. Making the field ofview at least as large as the tag plus three macrodot units guaranteesthat pulse-position modulated values can be coherently sampled.

The only exceptions are the translation codes described in the nextsection, which are four macrodot units long. However, these are highlyredundant and the loss of up to four symbols at the edge of the field ofview is not a problem.

2.6 Encoded Codes and Codewords

In this following section (Section 2.6), each symbol in FIGS. 10 to 14is shown with a unique label. The label consists of an alphabetic prefixwhich identifies which codeword the symbol is part of, and a numericsuffix which indicates the index of the symbol within the codeword. Forsimplicity only data symbols 304 are shown, not registration symbols.

Although some symbol labels are shown rotated to indicate the symmetryof the layout of certain codewords, the layout of each symbol isdetermined by its position within a symbol group and not by the rotationof the symbol label (as described in, for example, the Applicant's USPublication No. 2006/146069).

2.6.1 Registration Symbols

Each registration symbol is encoded using 2-6PPM. FIG. 9 shows thelayout of the registration symbol.

As shown in FIG. 6, the horizontal and vertical registration symbolseach appear once within a symbol group. The registration symbols of anentire tag typically indicate the vertical and horizontal translation ofthe tag by coding two orthogonal translation codes, and the orientationof the tag by coding two orthogonal direction codes.

Each registration symbol also encodes a one-bit symbol of a flag code(see Section 2.6.2).

Table 3 defines the mapping from 2-6PPM registration symbol values toflag code, direction code and translation code symbol values.

TABLE 3 2-6 PPM registration symbol values to flag code, direction codeand translation code symbol mapping flag code direction translation 2-6PPM symbol symbol code symbol code symbol value {p₅-p₀} value valuevalue 001,001 0 0 0 000,011 1 100,010 0 1 011,000 1 001,010 0 0 1000,101 1 010,100 0 1 101,000 1 010,001 0 0 2 000,110 1 100,100 0 1110,000 1 001,100 unused 010,010 100,001

Each row of symbol groups and each column of symbol groups encodes athree-symbol 3-ary cyclic position code. (The Applicant's cyclicposition codes are described in U.S. Pat. No. 7,082,562, the contents ofwhich is herein incorporated by reference). The code consists of thecodeword (0, 1, 2) and its cyclic shifts. The code has a minimumdistance of 3, allowing a single symbol error to be corrected. The codesof an entire tag form a code with a minimum distance of 9, allowing 4symbol errors to be corrected. If additional symbols are visible withinthe field of view then they can be used for additional redundancy.

The translation code symbol in the middle of the codeword (i.e. 1) ismapped to a set of 2-6PPM symbol values that are each other's reverse,while the two translation code symbols at the ends of the codeword (i.e.0 and 2) are each mapped to a set of 2-6PPM symbol values that are thereverses of the 2-6PPM symbol values in the other set. Thus a 0 readupside-down (i.e. rotated 180 degrees) becomes a 2, and vice versa,while a 1 read upside-down remains a 1. This allows translation to bedetermined independently of rotation.

Each 2-6PPM symbol value and its reverse map to opposite direction codesymbol values. The vertical registration symbols of an entire tag encode9 symbols of a vertical direction code. This has a minimum distance of9, allowing 4 symbol errors to be corrected. The horizontal registrationsymbols of an entire tag encode 9 symbols of a horizontal directioncode. This has a minimum distance of 9, allowing 4 symbol errors to becorrected. If additional symbols are visible within the field of viewthen they can be used for additional redundancy. Any erasures detectedduring decoding of a translation code can also be used during decodingof a direction code, and vice versa. Together the orthogonal directioncodes allow the orientation of the tag to be determined.

The top left corner of an un-rotated tag is identified by a symbol groupwhose translation symbols are both zero and whose direction symbols areboth zero.

2.6.2 Flag Code

The flag symbol consists of one bit of data, and is encoded in eachvertical and horizontal registration symbol, as shown in Table 3.

The flag symbol is unique to a tag and is therefore coded redundantly ineach quadrant of the tag. Since the flag symbol is encoded in eachregistration symbol, it appears eight times within each quadrant. Eightsymbols form a code with a minimum distance of 8, allowing 3 errors tobe corrected. If additional symbols are visible within the field of viewthen they can be used for additional redundancy. Any erasures detectedduring decoding of translation and/or direction codes can also be usedduring decoding of the flag code, and vice versa.

2.6.3 Coordinate Data

The tag contains an x-coordinate codeword and a y-coordinate codewordused to encode the x and y coordinates of the tag respectively. Thecodewords are of a shortened 2⁴-ary (10, p) Reed-Solomon code, where pcan vary from 2 to 5. The tag therefore encodes between 8 and 20 bits ofinformation for each coordinate. This reduces to 7.8 to 19.5 bits oncebase-15 conversion occurs.

Each x coordinate codeword is replicated twice within the tag—in eachhorizontal half (“north” and “south”), and is constant within the columnof tags containing the tag. Likewise, each y coordinate codeword isreplicated twice within the tag—in each vertical half (“east” and“west”), and is constant within the row of tags containing the tag. Thisguarantees that an image of the tag pattern large enough to contain acomplete tag is guaranteed to contain a complete instance of eachcoordinate codeword, irrespective of the alignment of the image with thetag pattern. The instance of either coordinate codeword may consist offragments from different tags.

It should be noted that, in the present invention, some coordinatesymbols are not replicated and are placed on the dividing line betweenthe two halves of the tag. This arrangement saves tag space since thereare not two complete replications of each x-coordinate codeword and eachy-coordinate codeword contained in a tag. Since the field of view is atleast three macrodot units larger than the tag (as discussed in Section2.10), the coordinate symbols placed on the dividing line (having awidth 2 macrodot units) are still captured when the surface is imaged.Hence, each interaction with the coded surface still provides the taglocation.

The layout of the x-coordinate codeword is shown in FIG. 10. The layoutof the y-coordinate codeword is shown in FIG. 11. It can be seen thatx-coordinate symbols X4, X5, X6, X7, X8 and X9 are placed in a centralcolumn 310 of the tag 4, which divides the eastern half of the tag fromthe western half. Likewise, the y-coordinate symbols Y4, Y5, Y6, Y7, Y8and Y9 are placed in a central row 312 of the tag 4, which divides thenorthern half of the tag from the southern half.

The central column 310 and central row 312 each have a width q, whichcorresponds to a width of 2s, where s is the macrodot spacing.

2.6.4 Common Data

The tag contains four codewords A, B, C and D which encode informationcommon to a set of contiguous tags in a surface region. The codewordsare of a 2⁴-ary (15, 9) Reed-Solomon code. The tag therefore encodes upto 144 bits of information common to a set of contiguous tags. Thisreduces to 140 bits once base-15 conversion occurs.

The common codewords are replicated throughout a tagged region. Thisguarantees that an image of the tag pattern large enough to contain acomplete tag is guaranteed to contain a complete instance of each commoncodeword, irrespective of the alignment of the image with the tagpattern. The instance of each common codeword may consist of fragmentsfrom different tags.

The layout of the common codewords is shown in FIG. 12. The codewordshave the same layout, rotated 90 degree relative to each other.

2.6.5 Optional Data

The tag optionally contains a codeword E. This codeword may be used toencode a secret-key signature or a fragment of an embedded data object.These are discussed further in Sections 2.9.4 and Section 2.9.3respectively. The codeword is of a 2⁴-ary (15, 9) Reed-Solomon code.

The layout of the optional codeword is shown in FIG. 13.

2.6.6 Secret-Key Signature

The tag optionally contains an entire secret-key digital signaturecommon to a set of contiguous tags in a surface region. The signatureconsists of sixteen 2⁴-ary symbols (i.e. symbol E15 is also used). Thetag therefore optionally encodes up to 64 bits of secret-key signaturedata.

The signature is replicated throughout a tagged region. This guaranteesthat an image of the tag pattern large enough to contain a complete tagis guaranteed to contain a complete instance of the signature,irrespective of the alignment of the image with the tag pattern. Theinstance of the signature may consist of fragments from different tags.

The signature, if present, is encoded in the E codeword described inSection 2.6.5. Digital signatures are discussed further in Section2.9.4.

2.6.7 Complete Tag

FIG. 14 shows the layout of the data of a complete tag, with each symbolgroup comprising ten data symbols. The vertical and horizontalregistration symbols are not shown in FIG. 14.

2.7 Reed-Solomon Encoding 2.7.1 Reed-Solomon Codes

All data is encoded using a Reed-Solomon code defined over GF(2⁴). Thecode has a natural length n of 15. The dimension k of the code is chosento balance the error correcting capacity and data capacity of the code,which are (n−k)/2 and k symbols respectively.

The code may be punctured, by removing high-order redundancy symbols, toobtain a code with reduced length and reduced error correcting capacity.The code may also be shortened, by replacing high-order data symbolswith zeros, to obtain a code with reduced length and reduced datacapacity. Both puncturing and shortening can be used to obtain a codewith particular parameters. Shortening is preferred, where possible,since this avoids the need for erasure decoding.

The code has the following primitive polynominal:

p(x)=x ⁴ +x+1

The code has the following generator polynominal:

${g(x)} = {\prod\limits_{i = 1}^{n - k}\; \left( {x + \text{?}} \right)}$?indicates text missing or illegible when filed

For a detailed description of Reed-Solomon codes, refer to Wicker, S. B.and V. K. Bhargava, eds., Reed-Solomon Codes and Their Applications,IEEE Press, 1994.

2.7.2 Codeword Organization

As shown in FIG. 15, redundancy coordinates r_(i) and data coordinatesd_(i) of the code are indexed from left to right according to the powerof their corresponding polynomial terms. The symbols X_(i) of a completecodeword are indexed from right to left to match the bit order of thedata. The bit order within each symbol is the same as the overall bitorder.

2.7.3 Code Instances

Table 4 defines the parameters of the different codes used in the tag.

TABLE 4 Codeword instances data length dimension error-correctingcapacity^(a) name description (n) (k) capacity (symbols) (bits) X, Ycoordinate codewords  10^(b) 5 2 19.5 (see Section 2.6.3) 4 3 15.6 3 311.7 2 4 7.8 A, B, C, common codewords 15 9 3 35 D (see Section 2.6.4) Eoptional codeword 15 9 3 35 (see Section 2.6.5) ^(a)takes into accountsymbol-wise conversion to base 15 to allow 2-6 PPM encoding^(b)shortened

2.8 Tag Coordinate Space

The tag coordinate space has two orthogonal axes labeled x and yrespectively. When the positive x axis points to the right then thepositive y axis points down.

The surface coding does not specify the location of the tag coordinatespace origin on a particular tagged surface, nor the orientation of thetag coordinate space with respect to the surface. This information isapplication-specific. For example, if the tagged surface is a sheet ofpaper, then the application which prints the tags onto the paper mayrecord the actual offset and orientation, and these can be used tonormalise any digital ink subsequently captured in conjunction with thesurface.

The position encoded in a tag is defined in units of tags. Byconvention, the tag position is taken to be the position of the top lefttarget in each tag.

2.9 Tag Information Content 2.9.1 Field Definitions

Table 5 defines the information fields embedded in the surface coding.

TABLE 5 Field Definitions width field (bits) description unique to tagactive area flag 1 A flag indicating whether the area^(a) immediatelysurrounding a tag intersects an active area. x coordinate 7.8-19.5 Theunsigned x coordinate of the tag^(b). y coordinate 7.8-19.5 The unsignedy coordinate of the tag^(b). common to tagged region encoding format 4The format of the encoding. 0: the present encoding. Other values arereserved region flags 12 Flags controlling the interpretation of regiondata (see Table 6). coordinate precision 2 A value (p) indicating theprecision of x and y coordinates according to the formula 8 + 4p.macrodot size ID 4 The ID of the macrodot size. coordinate width ID 2The ID of the width (w) of the x and y coordinates. 0: 7.8 bits 1: 11.7bits 2: 15.6 bits 3: 19.5 bits region ID space ID 6 The ID of the regionID space. 0: Netpage 1: EPC Other values are reserved for future use.region ID 96 The ID of the region containing the tags. secret-keysignature 64 An optional secret-key signature of the region. CRC (CyclicRedundancy 16 A CRC^(c) of common tag data. Check) ^(a)the diameter ofthe area, centered on the tag, is nominally 2.5 times the diagonal sizeof the tag; this is to accommodate the worst-case distance between thenib position and the imaged tag ^(b)allows a coordinate value range of857 mm (large enough for an A1 sheet) to 28.9 km for the nominal tagsize of 3.81 mm (based on the nominal macrodot size and 30 macrodots pertag) ^(c)CCITT CRC-16 [see ITU, Interface between Data TerminalEquipment (DTE) and Data Circuit-terminating Equipment (DCE) forterminals operating in the packet mode and connected to public datanetworks by dedicated circuit, ITU-T X.25 (10/96)], computed in bitorder on raw codeword data (see Table 4).

An active area is an area within which any captured input should beimmediately forwarded to the corresponding Netpage server 10 forinterpretation. This also allows the Netpage server 10 to signal to theuser that the input has had an immediate effect. Since the server hasaccess to precise region definitions, any active area indication in thesurface coding can be imprecise so long as it is inclusive.

TABLE 6 Region flags bit meaning 0 Region is interactive, i.e. x andy-coordinates are present. 1 Region is active, i.e. the entire region isan active area. Otherwise active areas are identified by individualtags' active area flags. 2 Region ID is not serialized^(a). 3 Region hassecret-key signature (see Section 2.9.4) 4 Region has embedded data. 5Embedded data is a public-key signature (see Sections 2.9.3 and 2.9.4).6 Page description is associated with region is public. Otherwise pagedescription is private. other Reserved for future use. Must be zero.^(a)For an EPC this means that the serial number is replaced by a layoutnumber, to allow the package design associated with a product to varyover time (see US 2007/0108285, the contents of which is hereinincorporated by reference).

2.9.2 Mapping of Fields to Codewords

Table 7 defines how the information fields map to codewords.

TABLE 7 Mapping of fields to codewords codeword codeword bits fieldwidth field bits X w-1:0  x coordinate w all Y w-1:0  y coordinate w allA 15:0  CRC^(a) 16 all 34:16 region ID 19 18:0 B 3:0 encoding format 4all 15:4  region flags 12 all 19:16 macrodot size ID 4 all 21:20coordinate width ID 2 all 27:22 region ID space ID 6 all 34:28 region ID7 25:19 C 34:0  region ID 35 60:26 D 34:0  region ID 35 95:61 E all datafragment 35 all E all^(b) secret-key signature 64 all ^(a)the CRC iscomputed in bit order on the data portions of the A, B, C and Dcodewords, in that order, excluding the CRC field itself ^(b)entirecodeword is used for data i.e. there is no redundancy

As shown in Table 7, codeword E either contains a data fragment or asecret-key signature. These are described in Section 2.9.3 and Section2.9.4 respectively. The secret-key signature is present in a particulartag if the “region has secret-key signature” flag in the region flags isset, and the tag's active area flag is set. The data fragment is presentif the “region contains embedded data” flag in the region flags is setand the tag's active area flag is not set.

When the region flags indicate that a particular codeword is absent thenthe codeword is not coded in the tag pattern, i.e. there are nomacrodots representing the codeword. This applies to the X, Y and Ecodewords.

2.9.3 Embedded Data Object

If the “region contains embedded data” flag in the region flags is setthen the surface coding contains embedded data. The embedded data isencoded in multiple contiguous tags' data fragments, and is replicatedin the surface coding as many times as it will fit.

The embedded data is encoded in such a way that a random and partialscan of the surface coding containing the embedded data can besufficient to retrieve the entire data. The scanning system reassemblesthe data from retrieved fragments, and reports to the user whensufficient fragments have been retrieved without error.

As shown in Table 8, each block has a data capacity of 170-bits. Theblock data is encoded in the data fragments of a contiguous group of sixtags arranged in a 3×2 rectangle.

The block parameters are as defined in Table 8. The E codeword of eachtag may encode a fragment of the embedded data.

TABLE 8 Block parameters parameter value description w 3 The width ofthe block, in tags h 2 The height of the block, in tags. b 170 The datacapacity of the block, in bits

If the E codeword of a particular tag does not contain a fragment of theembedded data, then the pen 101 can discover this implicitly by thefailure of the codeword to decode, or explicitly from the tag's activearea flag.

Data of arbitrary size may be encoded into a superblock consisting of acontiguous set of blocks, typically arranged in a rectangle. The size ofthe superblock may be encoded in each block.

The superblock is replicated in the surface coding as many times as itwill fit, including partially along the edges of the surface coding.

The data encoded in the superblock may include, for example, moreprecise type information, more precise size information, and moreextensive error detection and/or correction data.

2.9.4 Digital Signatures

As described in Section 2.6.6, a region may contain a digital signature.

If the <region has a secret-key signature> flag in the region flags isset, then the region has a secret-key digital signature. In an onlineenvironment the secret-key signature can be verified, in conjunctionwith the region ID, by querying a server with knowledge of thesecret-key signature or the corresponding secret key.

If the region contains embedded data and the <embedded data is apublic-key signature> flag in the region flag is set, then the surfacecoding contains an embedded public-key digital signature of the regionID.

In an online environment any number of signature fragments can be used,in conjunction with the region ID and optionally the secret-keysignature, to validate the public-key signature by querying a serverwith knowledge of the full public-key signature or the correspondingprivate key.

In an offline (or online) environment the entire public-key signaturecan be recovered by reading multiple tags, and can then be verifiedusing the corresponding public signature key. The actual length and typeof the signature are determined from the region ID during signaturevalidation i.e. typically from a previously-retrieved digital signatureassociated with a sequence of region IDs.

Digital signature verification is discussed in the Applicant's USPublication No. 2007/0108285, the contents of which are hereinincorporated by reference.

2.10 Tag Imaging and Decoding

The minimum imaging field of view required to guarantee acquisition ofdata from an entire tag has a diameter of 46.7s (i.e. ((3×10)+3)√2s),allowing for arbitrary rotation and translation of the surface coding inthe field of view. Notably, the imaging field of view does not have tobe large enough to guarantee capture of an entire tag—the arrangement ofthe data symbols within each tag ensures that a any square portion oflength (l+3s) captures the requisite information in full, irrespectiveof whether a whole tag is actually visible in the field-of-view. As usedherein, l is defined as the length of a tag.

In terms of imaging the coding pattern, the imaging field-of-view istypically a circle. Accordingly, the imaging field-of-view shouldpreferably have diameter of at least (l+3s)√2 and less than two tagdiameters. Importantly, the field-of-view is not required to be at leasttwo tag diameters, in contrast with prior art tag designs, because it isnot essential in the present invention to capture an entire tag in thefield of view.

The extra three macrodot units ensure that pulse-position modulatedvalues can be decoded from spatially coherent samples. Furthermore, theextra three macrodot units ensure that all requisite data symbols can beread with each interaction. These include the coordinate symbols from acentral column or row of a tag (see Section 2.6.3) having a width of 2s,and data symbols 304A extending from the perimeter edges of each tag byone macrodot unit (1s).

In the present context, a “tag diameter” is given to mean the length ofa tag diagonal.

Given a maximum macrodot spacing of 127 microns, this gives a requiredfield of view of 5.93 mm.

FIG. 16 shows a tag image processing and decoding process flow up to thestage of sampling and decoding the data codewords. Firstly, a raw image802 of the tag pattern is acquired (at 800), for example via an imagesensor such as a CCD image sensor, CMOS image sensor, or a scanninglaser and photodiode image sensor. The raw image 802 is then typicallyenhanced (at 804) to produce an enhanced image 806 with improvedcontrast and more uniform pixel intensities. Image enhancement mayinclude global or local range expansion, equalization, and the like. Theenhanced image 806 is then typically filtered (at 808) to produce afiltered image 810. Image filtering may consist of low-pass filtering,with the low-pass filter kernel size tuned to obscure macrodots 302 butto preserve targets 301. The filtering step 808 may include additionalfiltering (such as edge detection) to enhance target features 301.Encoding of data codewords 304 using pulse position modulation (PPM)provides a more uniform coding pattern 3 than simple binary dot encoding(as described in, for example, U.S. Pat. No. 6,832,717). Advantageously,this helps separate targets 301 from data areas, thereby allowing moreeffective low-pass filtering of the PPM-encoded data compared tobinary-coded data.

Following low-pass filtering, the filtered image 810 is then processed(at 812) to locate the targets 301. This may consist of a search fortarget features whose spatial inter-relationship is consistent with theknown geometry of the tag pattern. Candidate targets may be identifieddirectly from maxima in the filtered image 810, or may be the subject offurther characterization and matching, such as via their (binary orgrayscale) shape moments (typically computed from pixels in the enhancedimage 806 based on local maxima in the filtered image 810), as describedin U.S. Pat. No. 7,055,739, the contents of which is herein incorporatedby reference.

The identified targets 301 are then assigned (at 816) to a target grid818. Each cell of the grid 818 contains a symbol group 303, and severalsymbol groups will of course be visible in the image. At this stage,individual tags 4 will not be identifiable in the target grid 818, sincethe targets 301 do not demarcate one tag from another.

To allow macrodot values to be sampled accurately, the perspectivetransform of the captured image must be inferred. Four of the targets301 are taken to be the perspective-distorted corners of a square ofknown size in tag space, and the eight-degree-of-freedom perspectivetransform 822 is inferred (at 820), based on solving the well-understoodequations relating the four tag-space and image-space point pairs.Calculation of the 2D perspective transform is described in detail in,for example, Applicant's U.S. Pat. No. 6,832,717, the contents of whichis herein incorporated by reference.

Since each image will typically contain at least 16 targets arranged ina square grid, the accuracy of calculating the 2D perspective transformis improved compared to the Applicant's previous tag designs describedin, for example, U.S. Pat. No. 6,832,717. Hence, more accurate positioncalculation can be achieved with the tag design of the presentinvention.

The inferred tag-space to image-space perspective transform 822 is usedto project each known macrodot position in tag space into image space.Since all bits in the tags are represented by PPM-encoding, the presenceor absence of each macrodot 302 can be determined using a localintensity reference, rather than a separate intensity reference. Thus,PPM-encoding provides improved data sampling compared with pure binaryencoding.

The next stage determines the translation and orientation of the tag(s),or portions thereof, in the field of view relative to the target grid818. Two or more orthogonal registration symbols (‘VRS’ and ‘HRS’) aresampled (at 824), to allow decoding of the orthogonal translationcodewords and the orthogonal direction codewords.

Decoding of two or more orthogonal translation codewords (at 828) isused to determine the translation 830 of tags(s) in the field of viewrelative to the target grid 818. This enables alignment of the tags 4with the target grid 818, thereby allowing individual tag(s), orportions thereof, to be distinguished in the coding pattern 3 in thefield of view. Since each symbol group 303 contains orthogonalregistration symbols, multiple translation codes can be decoded toprovide robust translation determination. As described in Section 2.6.1,the translation code is a cyclic position code. Since each row and eachcolumn of a tag contains M symbol groups, the code has minimum distanceM×M. This allows very robust determination of the alignment of tags 4with the target grid 818. The alignment needs to be both robust andaccurate since there are many possible alignments when each tag 4contains multiple symbol groups 303.

Likewise, at least two orthogonal direction codes are decoded (at 825)to provide the orientation 826. As described in Section 2.6.1, since Nvertical registration symbols in a tag form a vertical direction codewith minimum distance N, the vertical direction code is capable ofcorrecting (N−1)/2 errors. The horizontal direction code is similarlycapable of correcting (N−1)/2 errors using N horizontal registrationsymbols Hence, orientation determination is very robust and capable ofcorrecting errors, depending on the number of registration symbolssampled.

Once initial imaging and decoding has yielded the 2D perspectivetransform, the orientation, and the translation of tag(s) relative tothe target grid, the data codewords 304 can then be sampled and decoded(at 836) to yield the requisite decoded codewords 838.

Decoding of the data codewords 304 typically proceeds as follows:

-   -   sample common Reed-Solomon codewords    -   decode common Reed-Solomon codewords    -   verify common tag data CRC    -   on decode error flag bad region ID sample    -   determine encoding type, and reject unknown encoding    -   determine region flags    -   determine region ID    -   determine x and y coordinate widths from coordinate width ID    -   sample and decode x and y coordinate Reed-Solomon codewords    -   determine tag x-y location from codewords    -   determine nib x-y location from tag x-y location and perspective        transform taking into account macrodot size (from macrodot size        ID)    -   sample and decode four or more flag symbols to determine active        area flag    -   determine active area status of nib location with reference to        active area flag    -   encode region ID, nib x-y location, and nib active area status        in digital ink (“interaction data”)    -   route digital ink based on region flags

The skilled person will appreciate that the decoding sequence describedabove represents one embodiment of the present invention. It will, ofcourse, be appreciated that the interaction data sent from the pen 101to the netpage system may include other data e.g. digital signature (seeSection 2.9.4), pen mode (see US 2007/125860), orientation data, pen ID,nib ID etc.

An example of interpreting interaction data, received by the netpagesystem from the netpage pen 101, is discussed briefly above. A moredetailed discussion of how the netpage system may interpret interactiondata can be found in the Applicant's previously-filed applications (see,for example, US 2007/130117 and US 2007/108285, the contents of whichare herein incorporated by reference).

3. Netpage Pen 3.1 Introduction and Functional Overview

The Netpage pen 400 is a motion-sensing writing instrument which worksin conjunction with a tagged Netpage surface (see Section 2). TheNetpage pen 400 typically includes a conventional ballpoint pencartridge for marking the surface, an image sensor and processor forcapturing the absolute path of the pen on the surface and identifyingthe surface, a force sensor for simultaneously measuring the forceexerted on the nib, an optional Gesture button for indicating that aGesture is being captured, and a real-time clock for simultaneouslymeasuring the passage of time.

During normal operation, the Netpage pen 400 regularly samples theencoding of a surface as it is traversed by the Netpage pen's nib. Thesampled surface encoding is decoded by the Netpage pen to yield surfaceinformation comprising the identity of the surface, the absoluteposition of the nib of the Netpage pen on the surface, and the pose ofthe Netpage pen relative to the surface. The Netpage pen alsoincorporates a force sensor that produces a signal representative of theforce exerted by the nib on the surface.

Each stroke is delimited by a pen down and a pen up event, as detectedby the force sensor. Digital Ink is produced by the Netpage pen as thetimestamped combination of the surface information signal, force signal,and the Gesture button input. The Digital Ink thus generated representsa user's interaction with a surface—this interaction may then be used toperform corresponding interactions with applications that havepre-defined associations with portions of specific surfaces. (Ingeneral, any data resulting from an interaction with a Netpage surfacecoding is referred to herein as “interaction data”).

Digital Ink is ultimately transmitted to the Netpage server 10, butuntil this is possible it may be stored within the Netpage pen'sinternal non-volatile memory. Once received by a Netpage server 10, theDigital Ink may be subsequently rendered in order to reproduce usermarkup of surfaces such as annotations or notes, or to performhandwriting recognition. A category of Digital Ink known as a Gesturealso exists that represents a set of command interactions with asurface. (Although the Netpage server 10 is typically remote from thepen 400 as described herein, it will be appreciated that the pen mayhave an onboard computer system for interpreting Digital Ink).

The pen 400 incorporates a Bluetooth radio transceiver for transmittingDigital Ink to a Netpage server 10, usually via a relay device 601 (seeFIG. 4). When operating offline from a Netpage server, the pen bufferscaptured Digital Ink in non-volatile memory. When operating online to aNetpage server the pen transmits Digital Ink in real time as soon as allpreviously buffered Digital Ink has been transmitted.

The Netpage pen is supplied with a charging cradle referred to as aNetpage pen cradle 426 (see FIG. 19). The Netpage pen cradle 426contains a Bluetooth to USB relay and connects via a USB cable to acomputer which provides communications support for local applicationsand access to Netpage services.

The Netpage pen is powered by a rechargeable battery. The battery is notaccessible to or replaceable by the user. Power to charge the Netpagepen is usually sourced from the Netpage pen cradle, which in turn cansource power either from a USB connection, or from an external ACadapter.

The Netpage pen's nib is user retractable, which serves the dual purposeof protecting surfaces and clothing from inadvertent marking when thenib is retracted, and signalling the Netpage pen to enter or leave apower-saving state when the nib is correspondingly retracted orextended.

3.2 Ergonomics and Layout

The overall weight (40 g), size and shape (155 mm×19.8 mm×18 mm) of theNetpage pen 400 fall within the bounds of conventional handheld writinginstruments.

Referring to FIG. 1, a rounded profile gives the pen an ergonomicallycomfortable shape to grip when the Netpage pen 400 is used in thecorrect functional orientation. It is also a practical shape foraccommodating the internal components.

A user typically writes with the Netpage pen 400 at a nominal pitch ofabout 30 degrees from the normal toward the hand when held (positiveangle) but seldom operates the Netpage pen at more than about 10 degreesof negative pitch (away from the hand). The range of pitch angles overwhich the Netpage pen is able to image the pattern on the paper has beenoptimized for this asymmetric usage. The shape of the Netpage penassists with correct orientation in a user's hand.

One or more colored user feedback LEDs 420 (see FIGS. 1, 21 and 39)illuminate corresponding indicator window(s) 421 on the upper surface ofthe Netpage pen 400. The indicator window(s) 421 are positioned toremain unobscured when the Netpage pen 400 is held in a typical writingposition.

Referring now to FIG. 17, a ballpoint pen cartridge 402 is housed in anupper portion of the Netpage pen's housing 404, placing it consistentlywith respect to the user's grip and providing good user visibility ofthe nib 406 whilst the Netpage pen 400 is in use. The space below theballpoint pen cartridge 402 is used for the main PCB 408 (which issituated in the centre of the Netpage pen 400) and for the battery 410(which is situated in the base of the Netpage pen). As shown in FIG. 2,the tag-sensing optics 412 are placed unobtrusively below the nib (withrespect to nominal pitch).

The ballpoint pen cartridge 402 is front-loading to simplify coupling toan internal force sensor 442.

Still referring to FIG. 2, the nib molding 414 of the Netpage pen 400 isswept back below the ballpoint pen cartridge 402 to prevent contactbetween the nib molding and the paper surface when the Netpage pen isoperated at maximum pitch. The Netpage pen's optics 412 and a pair ofnear-infrared illumination LEDs 416 are situated behind a filter window417 (see FIG. 22) located below the nib—the Netpage pen's imaging fieldof view emerges through this window, and the illumination LEDs alsoshine through this window. The use of two illumination LEDs 416 ensuresa uniform illumination field. The LEDs can also be controlledindividually so as to allow dynamic avoidance of undesirable reflectionswhen the Netpage pen is held at some angles, especially on glossy paper.

3.3 Netpage Pen Feedback Indications

The Netpage pen 400 may incorporate one or more visual user indicators420 that are used to convey the pen status to a user, such as batterystatus, online status and/or capture blocked status. Each indicator 420illuminates a shaped aperture or diffuser in the Netpage pen's housing404—the shape of the aperture or diffuser is typically an icon thatcorresponds to the nature of the indication. An additional batterystatus indicator used to indicate charging state is also visible fromthe top-rear of the Netpage pen whilst the pen is inserted in to theNetpage pen cradle.

An optional battery status indicator typically comprises a red and agreen LED and provides feedback on remaining battery capacity andcharging state to a user. An optional online status indicator typicallycomprises a green LED which provides feedback on the state of aconnection to a Netpage server, and also provides feedback duringBluetooth pairing operations.

3.3.1 Capture Blocked Indicator

The capture blocked indicator comprises a red LED and provides errorfeedback when Digital Ink capture is blocked. There may be a number ofconditions under which the Netpage pen 400 is incapable of capturingdigital ink, or is incapable of capturing digital ink of adequatequality.

For example, the pen 400 may be unable to capture (adequate quality)digital ink from a surface because it is unable to image the tag patternon the surface or decode the imaged tag pattern. This may occur under anumber of conditions:

-   -   the surface is not tagged    -   the pen's field of view is slightly or fully off the edge of the        tagged surface    -   the tag pattern is poorly printed (e.g. due to printing errors,        or to the use of a poor-quality print medium)    -   the tag pattern is damaged (e.g. the tag pattern is faded or        smeared, or the surface is scratched or dirty)    -   the tag pattern is counterfeit (i.e. it contains an invalid        digital signature)    -   the pen's tilt is excessive (i.e. causing excessive geometric        distortion, defocus blur and/or poor illumination)    -   the pen's speed is excessive (i.e. causing excessive motion        blur)    -   the tag pattern is obscured by specular reflection (i.e. from        the surface itself or from the printed tag pattern or graphics)

The pen may be unable to store digital ink because its internal bufferis full.

The pen may also choose not to capture digital ink under a number ofcircumstances:

-   -   the pen is not registered (as indicated by the pen's own        internal record, or by the server)    -   the pen is not connected (i.e. to a server)    -   the pen has been blocked from capturing (e.g. on command from        the server)    -   the pen's user has not been authenticated (e.g. via a biometric        such as a fingerprint or handwritten signature or password)    -   the pen is stolen (i.e. as reported by the server)    -   the pen's ink cartridge is empty (e.g. the pen is a universal        pen as described in U.S. Pat. No. 6,808,330 incorporated herein        by reference, so its ink consumption is easily monitored)

The pen may also choose to not to capture digital ink if it detects aninternal hardware error, such as a malfunctioning force sensor.

The visual capture blocked indicator LED 420 typically indicates to theuser that digital ink capture is blocked, e.g. due to one of theconditions described above. This indicator LED 420 may also be used toindicate when capture is close to being blocked, such as when the tagpattern decoding rate drops below a threshold, or the tilt or speed ofthe pen becomes close to excessive, or when the pen's digital ink bufferis almost full.

3.3.2 Alternative Indicators

Although a single capture-blocked indicator LED 420 is exemplified inthe Netpage pen 400 described herein, there are, of course, a number ofways to indicate to the user that digital ink capture is blocked (or isclose to being blocked), such as auditory and haptic indications.

Visual indications may be characterized by their color, intensity, andspatial extent, possibly with spatial and temporal modulation. They maybe implemented using single or multiple LEDs and 2D displays (e.g. LCDs)etc.

Auditory indications are characterized by their frequency content andintensity, possibly with temporal modulation. They can be implementedusing buzzers and speakers etc.

Haptic indications stimulate the sense of touch, and in the presentcontext can be implemented using buzzers and vibrators. They arecharacterised by their frequency content and intensity, possibly withtemporal modulation.

A vibrator can consist of a small electric motor fitted with aneccentric weight, as used for vibrating alerts in mobile phones andsimilar devices.

If the pen has an auto-retracting nib, the pen can auto-retract the nibwhen it detects a condition that blocks capture. This can include allsuch conditions or just selected conditions (such as buffer full).

When the pen has an auto-extending nib the pen can inhibit nib extensionwhen it detects a condition that blocks capture. This can include allconditions that are detectable when nib extension is attempted, or justselected conditions (such as buffer full).

When the pen controls the marking capacity of the nib, e.g. as describedin U.S. Pat. No. 6,808,330, the pen can inhibit marking when it detectsa condition that blocks capture. This can include all such conditions orjust selected conditions (such as buffer full).

3.4 Netpage Pen Cradle 426

As shown in FIG. 18, the Netpage pen's cradle contacts 424 are locatedbeneath the nose cone 409. These contacts 424 connect with a set ofcorresponding contacts in the Netpage pen cradle 426 upon insertion, andare used for charging the Netpage pen 400.

FIG. 19 shows the Netpage pen 400 docked in the Netpage pen cradle 426.The Netpage pen cradle 426 is compact to minimise its desktop footprint,and has a weighted base for stability. Data transfer occurs between theNetpage pen 400 and the Netpage pen cradle 426 via a Bluetooth radiolink.

The Netpage pen cradle 426 may have two visual status indicators—a powerindicator, and an online indicator. The power indicator is illuminatedwhenever the Netpage pen cradle 426 is connected to a power supply—e.g.an upstream USB port, or an AC adapter. The online indicator providesfeedback when the Netpage pen 400 has established a connection to theNetpage pen cradle 426, and during Bluetooth pairing operations.

There are two main functions that are required by the Netpage pen cradle426:

-   -   provide a source of charge current so that the Netpage pen 400        can recharge its internal battery 410.    -   provide host communications Bluetooth wireless endpoint for the        Netpage pen 400 to connect to in order to ultimately communicate        with the Netpage server 10.

The Netpage pen cradle 426 has a built-in cable which ends in a singleUSB A-side plug for connecting to an upstream host.

In order to provide sufficient current for normal charging of theNetpage pen's battery 410, the Netpage pen cradle 426 is typicallyconnected to a root hub port, or a port on a self-powered hub. A secondoption for providing charging-only operation of the Netpage pen cradle426 is to connect the USB A-side plug to an optional AC adapter.

FIGS. 20A-D show the main charging and connection options for theNetpage pen 400 and Netpage pen cradle 426.

FIG. 20A shows a USB connection from a host (e.g. PC) to the Netpage pencradle 426. The Netpage pen 400 is seated in the Netpage pen cradle 426,and the Netpage pen cradle and the Netpage pen communicate wirelesslyvia Bluetooth. The Netpage pen cradle 426 is powered by a USB bus powerand the Netpage pen 400 is charged from the USB bus power. As a result,the maximum USB power of 500 mA must be available in order to charge thepen at the normal rate.

FIG. 20B shows a USB connection from a host (e.g. PC) to the Netpage pencradle 426. The Netpage pen 400 in use, and the cradle and pencommunicate wirelessly via Bluetooth. The Netpage pen cradle 426 ispowered by the USB bus power.

FIG. 20C shows an optional AC adapter connected to the Netpage pencradle 426. The Netpage pen 400 is seated in the Netpage pen cradle 426,and is charged from current supplied by the optional AC adapter.

FIG. 20D shows the Netpage pen in use. In this case, the Netpage pen iscommunicating to a host (e.g. PC) wirelessly using 3rd party Bluetoothwhich may be, for example, integrated into a laptop or mobile phone.

The Netpage pen cradle 426 contains a CSR BlueCore4 device. TheBlueCore4 device functions as a USB to Bluetooth bridge, and provides acompletely embedded Bluetooth solution.

3.5 Mechanical Design 3.5.1 Parts and Assemblies

Referring to FIGS. 21 and 22, the pen 400 has been designed as a highvolume product and has four major sub-assemblies:

an optical assembly 430;

a force sensing assembly 440 including force sensor 442;

a nib retraction assembly 460, which includes part of the force sensingassembly;

a main assembly 480, which includes the main PCB 408 and battery 410.

These assemblies and the other major parts can be identified in FIG. 21.As the form factor of the pen is to be as small as possible these partsare packed as closely as practical.

The pen housing 404, which defines the body of the pen, is comprised ofa pair of snap-fitting side moldings 403, a cover molding 405, anelastomer sleeve 407 and a nosecone molding 409. The cover molding 405includes one or more transparent windows 421, which provide visualfeedback to the user when the LEDs 420 are illuminated.

Although certain individual molded parts are thin walled (0.8 to 1.2 mm)the combination of these moldings creates a strong structure. The pen400 is designed not to be user serviceable and therefore the elastomersleeve 407 covers a single retaining screw 411 to prevent user entry.The elastomer sleeve 407 also provides an ergonomic high-frictionportion of the pen, which is gripped by the user's fingers during use.

3.5.2 Optical Assembly 430

The major components of the optical assembly 430 are as shown in FIGS.23 and 24. An optics PCB 431 has a rigid portion 434 and a flexibleportion 435. A ‘Himalia’ image sensor 432 is mounted on the rigidportion 434 of the optics PCB 431 together with an optics barrel molding438.

Since the critical positioning tolerance in the pen 400 is between theoptics and the image sensor 432, the rigid portion 434 of the optics PCB431 allows the optical barrel to be easily aligned to the image sensor.The optics barrel molding 438 has a molded-in aperture 439 near theimage sensor 432, which provides the location of a focusing lens 436.Since the effect of thermal expansion is very small on a molding of thissize, it is not necessary to use specialized materials.

The flexible portion 435 of the optics PCB 431 provides a connectionbetween the image sensor 432 and the main PCB 408. The flex is a 2-layerpolyimide PCB, nominally 75 microns thick, which allows somemanipulation during manufacture assembly. The flex 435 is L-shaped inorder to reduce its required bend radius, and wraps around the main PCB408. The flex 435 is specified as flex on install only, as it is notrequired to move after assembly of the pen. Stiffener is placed at theconnector (to the main PCB 408) to make it the correct thickness for theoptics flex connector 483A used on the main PCB (see FIG. 36B). Discretebypass capacitors are mounted onto the flex portion 435 of the opticsPCB 431. The flex portion 435 extends around the main PCB 408, andwidens to the rigid portion 434 at the image sensor.

The Himalia image sensor 432 is mounted onto the rigid portion 434 ofthe optics PCB 431 using a chip on board (COB) PCB approach. In thistechnology, the bare Himalia image sensor die 432 is glued onto the PCBand the pads on the die are wire-bonded onto target pads on the PCB. Thewire-bonds are then encapsulated to prevent corrosion. Four non-platedholes in the PCB next to the die 432 are used to align the PCB to theoptical barrel 438. The optical barrel 438 is then glued in place toprovide a seal around the image sensor 432. The horizontal positionaltolerance between the centre of the optical path and the centre of theimaging area on the image sensor die 432 is ±50 microns. In order to fitin the confined space at the front of the pen 400, the Himalia imagesensor die 432 is designed so that the pads required for connection inthe Netpage pen 400 are placed down opposite sides of the die.

3.5.3 Force Sensing Assembly 440 and Ballpoint Pen Cartridge 402

The force sensing assembly 440 is designed to accurately measure axialforce transmitted through the ballpoint pen cartridge 402 from the nib406 during use, both for tapping the surface of the paper and duringwriting. It is specified to sense between 0 and 500 gram force withenough fidelity to support handwriting recognition for Netpageapplications.

Whilst the force sensing assembly 440 is designed to be used inrecording the dynamic applied force for use in applications such assignature recognition and print rendering, its most common usage is as avery sensitive nib switch to detect the slightest contact of the nib 406with the paper. Therefore, the force sensing assembly 440 implicitlyencompasses a ‘nib switch’. When used as a nib switch, the nominalthreshold of sensitivity of the force sensing assembly is set at about10 g, and the resolution of the sensing at approximately 0.5 g. Thecombined mechanical and electrical bandwidth of the system is designedto allow time resolution of about 1 ms.

As described herein, the force sensing assembly 440 comprises a forcesensor 442, which may function as a nib switch having a predeterminedthreshold actuation force. However, it will be appreciated that, in analternative embodiment, the force sensor may be replaced with amechanical nib switch actuable by a low threshold force (e.g. 10 g).

Referring to FIG. 25, the complete force sensing assembly 440 comprisesthe replaceable ballpoint pen cartridge 402 (either marking ornon-marking), part of the nib retraction mechanism 460 which retains thepen cartridge, the preload spring 441 and the packaged force sensor 442.The force sensor 442 is connected to the main PCB 408 via a force sensorflex 443.

The ballpoint pen cartridge 402 has a first end comprising the nib 406and an opposite second end retained by a ratchet 444 (see FIG. 31). Theratchet 444 has a cartridge-retaining part (or ‘boot’) in the form of asheath 446, and a ratchet part comprising ratchet teeth 445. The secondend of the cartridge 402 is slidingly received inside the sheath 446 andis gripped, clamped or otherwise held so as to be retained securelytherein.

The ratchet 444 is a key component of the pen, which couples the forcesensing assembly 440 and the nib retraction mechanism 460. Itsratcheting function will be discussed in more detail below in connectionwith the nib retraction mechanism.

Referring to FIGS. 25, 31 and 32, the ratchet 444 is slidably receivedinside a barrel 448, which has a first abutment surface 490 and a secondabutment surface 491 defined in an inner wall thereof. The first andsecond abutment surfaces 490 and 491 are spaced apart and configured toprovide two stable positions for the cartridge 402 corresponding toextended and retracted nib positions, respectively. The ratchet 444 isurged towards either the first abutment surface 490 or the secondabutment surface 491 by means of a load spring 449. The load spring 449is engaged between the ratchet 444 and a spring retainer 450, which isfixed to the barrel. Hence, the load spring 449 ensures engagementbetween the ratchet 444 and the first abutment surface 490 in theextended position. The cartridge 402 slidingly travels through a centralaperture in the spring retainer 450 and a longitudinal axial void in theload spring 449 to be securely retained by the sheath part 446 of theratchet 444.

When the cartridge 402 is extended and in use, the barrel 448experiences an axial nib force via the ratchet 444 which is in abuttingengagement with the first abutment surface 490, as shown in FIG. 32. Theforce sensor 442 is coupled to the barrel 448 via a pin 451, which isintegrally molded with the barrel and extends from the barrel along anaxis parallel with the longitudinal cartridge axis. Hence, the pin 451transmits axial forces exerted on the barrel 448 by the cartridge 402(retained by the ratchet 444) to the force sensor 442, enabling theforce sensor to detect nib forces when the pen 400 is in use.

The force sensor 442 is typically a capacitive force sensor having afixed plate 467 and a moving plate 457. The pin 451 is engaged againstthe moving plate 457 of the force sensor 442, but is not fixed to themoving plate so as to avoid damage to the force sensor during removal ofthe cartridge 402. A preload spring 441 is arranged to urge the barrel448 (and thereby the pin 451) against the moving plate 457 of the forcesensor 442. The preload spring 441 therefore ensures engagement betweenthe pin and the moving plate, whilst obviating a fixed linkagetherebetween that could result in damage during cartridge removal.

Although the preload spring 441 provides some reaction to nib forces,the moving plate 457 of the force sensor is much stiffer than thepreload spring 441 and provides the dominant reaction force in the forcesensing system.

3.5.3.1 Force Sensor

The force sensor 442 operates on the principle of measuring thecapacitance between two parallel metal plates, which are resilientlyspaced apart. Since the capacitance between the plates can be measuredelectrically, the plate separation can be calculated. Further, since theplate separation is controlled by the applied force deflecting a spring,electrical measurement of the capacitance allows the force applied tothe nib 406 to be calculated. Typically, the plate separation is from100 to 500 microns, from 100 to 250 microns or from 100 to 200 microns.

Referring to FIG. 26, the force sensor is 442 is comprised of a basemoulding 454, a force sensing rigid/flex PCB 455 comprising the fixedplate 467 of the force sensor, a locating ring 452, a spring 456comprising the moving plate 457 of the force sensor, and a cap moulding458. The base and cap mouldings 454 and 458 are joined by fourheat-staking pins, and enclose the stackup of the force sensingrigid/flex PCB 455 and the spring 456. The exterior housing has aconductive coating which is connected to ground.

Referring to FIG. 27, the spring 456 is plated on the inside to providegood contact to the PCB 455. The spring 456 comprises an outer annularsupport 461, three spiral spring arms 459 and a moving plate 457 definedby an inner disk. The spring arms 459 may be cantilever and/or torsionsprings. When an axial force is applied to the moving plate 457, thespiral spring arms 459 flex allowing axial movement of the moving plateout of the plane of the outer annular support 461.

The spring 456 is usually manufactured from a single blank of resilientmaterial. Suitable materials include metals, such as stainless steel andberyllium-copper alloy. BeCu is highly suitable due its longevity.

It will be appreciated that flexing of the spiral spring arms 459results in a small rotation of the moving plate 457. However, there isno transmission of this rotation to the pin 451 as it is depressed dueto the low-friction surface of the pin. Once the axial force is removed,the spiral spring arms 459 return the moving plate 457 to its quiescentposition shown in FIG. 27, thereby providing a natural self-calibrationpoint. The moving plate 457 is typically made larger than the fixedplate 467 of the capacitor to ensure that the plates are always aligned.

The force sensing rigid/flex PCB 455 comprises a bonded stack of rigidand flexible PCBs, which provide both the fixed plate 467 and thedielectric for the parallel-plate capacitive force sensor. FIG. 28 showsschematically the various layers of the force sensing rigid/flex PCB455.

A 1.5 mm thick rigid PCB 462, comprised of a rigid FR4 layer 463 backedby a copper outer shield 464, provides a rigid base for the stack.

A first flexible PCB 465 is bonded to the rigid PCB 462 with a 12 micronadhesive layer (not shown). The first flexible PCB 465 comprises an 18micron copper layer 466, which provides the fixed plate 467 for theforce sensor, and a 12.5 micron polyimide layer 468. The PI layer 468provides the high barrier resistance required by the Jupiter forcesensor circuit. Polyimide also has a dielectric constant with very lowtemperature dependence, whilst being tough. Moreover, since polyimidehas a relative dielectric constant of 4, it also acts to increase thecapacitance of the force sensor.

A thicker second flexible PCB 469, having an aperture 470 punchedtherethrough, provides the air-gap of the force sensor. The secondflexible PCB 469 comprises a 127 micron thick PI layer 471 (which servesas a primary rigid spacer for the capacitor) and a copper layer 472,which provides a contact for the moving plate 457.

The PI layer 471 is bonded to the PI layer 468 of the first flexible PCB465, and the spring 456 sits on the upper surface of the PCB stack 455connected to an upper copper layer 472 of the second flexible PCB 469via a gold contact 473. The moving plate 457 is therefore able to moveinto the air-gap 470 and approach the fixed plate 467 by flexing of thespiral springs 459.

Three signals (fixed plate, moving plate, ground/shield) are takenoff-board as a cable using the first flex PCB 465. The cable providesessential shielding between the input and output signals and isexternally shielded by conducting ink layers screened over coverlays.

The construction of the plates with their PI spacer 471 ensures thatthey are able to withstand significant overforce without being damaged.The total distance between the moving plate 457 and the fixed plate 467is typically in the range of 100 to 200 microns, typically 150 to 180microns. This is provided the polyimide spacer layer 471 and the PIlayer 468, together with adhesive layers and the upper copper contactlayer 472.

In most capacitive force sensors, the dielectric itself provides aspring function allowing the moving plate to return to its zeroposition. However, such force sensors have relatively slowreturn-to-zero response times (about 100 ms or greater). The forcesensor 442 in the Netpage pen 400 is required to have a much fasterreturn-to-zero response time of less than 5 ms (preferably about 1 ms),in order to be able to capture pen strokes accurately. The Applicantshave found that such short return-to-zero response times can only beachieved with the force sensor 442 described above, having a movingplate spring 456 and rigid polyimide spacers.

A further advantage of the force sensor 442 is the dielectric comprisedof air 470 and polyimide 468. This provides a very high electricalbarrier resistance that is required between the plates in order tominimize capacitance measurement errors caused by leakage current. Afurther advantage of polyimide is that it has almost no temperaturecoefficient, thereby ensuring that the sensor capacitance does not driftwith temperature. Another advantage is that the polyimide layer 468protects the fixed plate 467 and ensures that the moving plate 457cannot contact the fixed plate.

3.5.3.2 Inverted Force Sensor

In the force sensor 442 described in Section 3.5.3.1, the moving plate457 moves towards the fixed plate 467 when an axial force is transmittedto the moving plate from the nib 406. It will be appreciated that it isequally possible to have an inverted force sensor arrangement wherebythe moving plate 457 moves away from the fixed plate 467, increasing thedielectric gap under an applied axial force. This arrangement has theadvantage that the capacitance is at a maximum when the applied force iszero, which improves the sensitivity of the system to small appliedforces.

FIG. 29 shows an inverted force sensor similar in construction to therigid/flex PCB stack 455 described above. Accordingly, like parts arelabeled similarly in FIG. 29. However, in the case of the inverted forcesensor, the rigid/flex PCB 455 has a hole punched through the centre ofthe PCB stack to allow the pin 451 to engage with the moving plate 457.In addition, the air gap between the moving plate 457 and the fixedplate 467 is reduced so that the capacitance at minimum applied force inthe inverted design is similar to the capacitance at maximum appliedforce in the non-inverted design.

3.5.3.3 Ballpoint Pen Cartridge 402

Referring to FIG. 30, the ballpoint pen cartridge 402 is a conventionalmedium ball (0.8-1.0 mm) ballpoint pen. It is typically supplied withblack ink, but blue ink is also possible. Although the Netpage pen 400has been described herein in an embodiment employing the ballpoint pencartridge 402, an alternative non-marking stylus cartridge in anidentical form-factor is also possible. Ink cartridges and styluscartridges may be used interchangeably in the Netpage pen 400 and thepresent invention is not limited to any particular type of cartridge.

The ballpoint pen cartridge design is similar in some respects to the“Cross” ballpoint pen cartridge, but the length is different and it isused in front-loading mode. The write-out of the ballpoint pen cartridgeis 2.5 km, similar to that used in high-usage conventional pens.

The nib style has been selected to optimize the optical clearance. Therigid tube is nickel plated brass 3.05 mm OD. The ink is a high qualityballpoint ink suitable for ISO 12757-2 documentary usage and compatiblewith this type of tube and nib. The tube is closed with a follower toprevent ink drawback. The ink is transparent in the near-infraredspectrum allowing the imaging system to see the Netpage surface-codingpattern printed underneath.

The cartridge 402 is designed to be user-replaceable. In order toextract the cartridge 402 (and referring now to FIGS. 22, 25 and 31),the nib 406 is pulled from the pen 400 using an extraction tool, therebycompressing the load spring 449 until the ratchet 444 abuts the springretainer 450, which acts as an end-stop for the ratchet. Engagement ofthe ratchet 444 with the spring retainer 450 causes the cartridge 402 tobe released from the sheath 446, allowing the cartridge 402 to beextracted. During cartridge removal, the barrel 448 is urged away fromthe force sensor 442 by abutment of the ratchet 444 with the springretainer 450. However, the preload spring 441 ensures that the barrel448 remains engaged with the force sensor 442 via the pin 451, therebyavoiding any potential damage to the force sensor.

A replacement cartridge 402 may be inserted into the pen by pushing thecartridge towards the ratchet 444, where it is retained by the sheath446. During insertion, the cartridge 402 travels through a plastic tube453, which guides the cartridge towards the ratchet 444 and prevents thecartridge from interfering with pen electronics. The plastic tube 453also creates a creepage barrier to EDS currents that fall on the exposednib 406 of the cartridge 402.

3.5.4 Nib Retraction Assembly 460

The Netpage pen's nib retraction assembly 460 allows the nib 406 to beretracted inside the body of the Netpage pen 400 so that it will notinadvertently mark clothing when it is, for instance, placed in a user'spocket.

Retracting the nib 406 also provides an input to the electronics thatallows the Netpage pen 400 to be put in a power saving state when thenib is retracted. Correspondingly, extending the nib 406 provides asignal to the electronics that causes the pen to be made ready forcapture of Digital Ink.

The nib retraction assembly 460 consists of seven parts as shown in FIG.31, the majority of which have already been described in connection withthe force sensing assembly. The ratchet 444 is a key component of thenib retraction assembly 460, having its ratchet-teeth 445 as well as thesheath 446 for retaining the cartridge 402. The ratchet 444 is slidablymovable within the barrel to provide the extended and retracted nibpositions. Referring to FIG. 32B, a series of longitudinal grooves 493defined in the inner surface of the barrel 448 guide the ratchet 444 bysliding engagement of the ratchet-teeth 445 in the grooves.

Referring to FIGS. 22, 31 and 32A, a plunger 474 is slidably movablewithin the barrel and has plunger-teeth 475, which slidingly engage inthe grooves 493 to guide the plunger towards the ratchet 444. Theplunger 474 may be moved forwards and backwards by actuation and releaseof a manually-operable button 476 coupled to the plunger. The plunger474 is engaged with the ratchet 444 via complementary abutting cammedend-surfaces on the ratchet-teeth 445 and the plunger-teeth 475. Thecamming engagement of these end-surfaces imparts a rotational force tothe ratchet 444 so that the ratchet rotates when it is pushed beyond alip of the grooves 493. Once the button 476 is released, the load spring449 pushes the ratchet 444 back towards the grooves 493, whereupon thecammed end-surfaces of the ratchet-teeth 445 abut with cammed lips ofthe grooves to impart further rotation to the ratchet. Accordingly,after one reciprocating actuation of the button 476, the ratchet 444 isrotated from a first set of grooves 493 into engagement with an adjacentsecond set of grooves. The first set of grooves guides the ratchet 444towards the first abutment surface 490 when the button 476 is released,while the second set of grooves allows the ratchet to slide towards thesecond abutment surface 491. The first and second sets of grooves 494are alternately defined around the inner surface of the barrel 448 sothat the ratchet 444 switches between extended and retracted positionsupon each actuation (and release) of the button 476. This arrangement issimilar to those typically found in standard retractable ballpoint pens.

As mentioned above, the button 476 is coupled to the plunger 474 duringactuation, enabling users to push the plunger 474 forwards inside thebarrel 448 and, hence, manually select alternate nib-retracted ornib-extended configurations for the pen 400. In order to prevent thebutton 476 transmitting any force (even very small forces) to the forcesensor 442 via the barrel 448 (thereby affecting nib forcemeasurements), a decoupling spring 479 biases the button 476 away fromcoupling engagement with the plunger 474. As shown in FIGS. 21 and 22,the decoupling spring 479 is engaged between a grommet 495, which isfixedly mounted to the pen housing 404, and the button 476. The grommet495 has a central aperture, which allows the plunger 474 to passtherethrough.

The barrel 448 is housed in a barrel housing 477, which is itselfsupported within the pen housing 404. The preload spring 441 is engagedbetween the barrel 448 and the barrel housing 477 so as to bias thebarrel, and thereby the pin 451, towards the moving plate 457 of theforce sensor 442.

3.5.4.1 Nib Extension/Retraction Detection

The Netpage pen 400 needs to ‘know’ the extension or retraction state ofthe cartridge 402 for a number of purposes. For example, if thecartridge is retracted, then the pen can initiate a quiescent or‘power-down’ state and thus conserve battery power, optionally afterpredetermined period of time has elapsed. Typically, the image sensor isdeactivated when the cartridge is retracted and/or the processor isconfigured not to generate Digital Ink using captured images.

Similarly, if the pen detects that the cartridge is extended, but theforce sensor 442 has not detected contact with a surface for apredetermined period of time, then the pen can enter a quiescent state.In addition, actuation of the button 476 may be used to initiate‘power-up’ behavior.

Referring to FIG. 22, the Netpage pen 400 has a switch 496 on the mainPCB 408, which is manually actuated by part of the button 476 when thebutton is pushed forwards by the user. Accordingly, actuation of theswitch 496 provides an input signal to the processor 500 indicating thatthe button 476 is being actuated for extension or retraction of thecartridge. Typically, actuation of the switch 496 is used to wake thepen up from a ‘power-down’ state to a ‘power-up’ state.

In addition, and still referring to FIG. 22, an optical sensor 497 onthe main PCB 408 is used to detect the final state (extended orretracted) of the cartridge 402 by sensing a dark patch painted on thecartridge. When the optical sensor 497 senses the dark patch, theprocessor 500 receives an input signal from the sensor indicating thatthe cartridge is extended. The plastic tube 453 may have a window toallow the optical sensor to sense the dark patch painted on thecartridge 402.

The pen's real-time clock may 515 be used in combination with the switch496 and optical sensor 497 to provide a further input signal to theprocessor. For example, the pen may be configured to initiate power-downonly after a predetermined period of time has elapsed, as determined bythe real-time clock 515.

3.5.4.2 Automatic Nib Retraction

As an alternative to (or in addition to) the manual nib retractionmechanism 460 described above, the Netpage pen 400 may incorporate anautomatic nib retraction feature.

There are situations where it is advantageous to retract the nib of aNetpage pen automatically, such as when the pen is incapable ofcapturing digital ink, or when the nib is in danger of being damaged.

There are a number of situations where the pen is incapable (orimminently incapable) of capturing digital ink, including:

-   -   when the surface is not coded    -   when the surface coding quality falls below a certain threshold    -   when the captured digital ink quality falls below a certain        threshold    -   when the digital ink buffer is full (or close to full)    -   when the remaining battery capacity is low    -   when there is no streaming connection to a Netpage server

There are a number of situations where the nib can be inferred to be indanger of being damaged, including:

-   -   when the pen is in freefall    -   when the pen experiences an impact shock    -   when the force sensor of the pen measures an excessive force

There are situations where it is advantageous to pre-emptively shut downthe pen and therefore pre-emptively retract the nib to prevent the userfrom inadvertently using it, including:

-   -   after an inactivity timeout, to conserve battery power    -   in the presence of extreme environmental conditions        (temperature, humidity, etc.)

There are situations where it is advantageous to retract the nib toinform the user about the operational limits of the pen, including:

-   -   excessive speed relative to a coded surface    -   excessive tilt relative to a coded surface

Ideally the pen's manual or automatic nib extension mechanism isinoperable whilst a nib-disablement condition continues to exist.

In general, automatic nib retraction is used to disable and/or protectthe nib. Alternative means for disabling and/or protecting the nibinclude automatic capping mechanisms (such as an auto-extending hood),or automatic disablement of the marking function of the nib. Forexample, U.S. Pat. No. 7,015,901, incorporated herein by reference,describes a pen whereby the marking function of the nib may be disabled.

The pen's automatic retraction mechanism may be triggered by the uservia a button coupled to a switch, or the pen may retain a manualretraction mechanism for this purpose.

In either case, nib retraction typically also signals the pen to enter alow-power state, i.e. the nib retraction button is the pen's ‘off’button. Automatically-triggered nib retraction also typically signalsthe pen to enter a low-power state.

If the pen incorporates an automatic extension mechanism (e.g. becauseit is the same mechanism as the retraction mechanism), then the user mayextend the nib via the same button as they use to retract the nib. Nibextension signals the pen to enter a power-up state where it isresponsive to pen-down interactions and is capable of digital inkcapture immediately after pen-down detection, i.e. the nib extensionbutton is the pen's ‘on’ button.

The automatic nib retraction mechanism may be implemented in a number ofdifferent ways. Many of these are also suitable for nib extension. Ingeneral the automatic nib retraction mechanism comprises a motor orother actuator which is coupled to the cartridge, or a boot (e.g.sheath) holding the cartridge, and which is controlled by softwarerunning on one of the pen's processors. The software is responsive tosignals from sensors in the pen, such as an accelerometer for free-falldetection, as well as its own internal state to trigger nib retraction.

Since the cartridge or boot is coupled to a nib switch or force sensor,the automatic retraction mechanism must not interfere with thiscoupling.

Referring to FIG. 33, there is shown schematically a first automaticretraction mechanism 600, which utilizes the same design as the manualretraction mechanism 460 described in Section 3.5.4. In the firstautomatic retraction mechanism 600, a motor 602 is coupled to theplunger 474 via a suitable coupling, such as a worm gear orrack-and-pinion coupling. The pen cartridge 402 (which may alternativelybe a stylus cartridge as discussed herein) is retained by the ratchet444, which itself engages with the barrel 448. Accordingly, movement ofthe plunger 474 results in nib retraction or extension.

Essentially, the motor 602 (and its coupling to the plunger 474) takesthe place of the manual retraction button 476. However, the firstautomatic retraction mechanism 600 may additionally include the manualretraction button 476, providing users with the option of either manualnib retraction/extension or automated nib retraction/extension. Whenmanual retraction/extension is required the motor 602 may be decoupledfrom the plunger 474.

Still referring to FIG. 33, the processor 500 is coupled to the motor602 and receives input signals to determine whether the nib should beextended or retracted. The input signal may be from a simple on/offswitch 604 or an accelerometer 606, which detects when the pen is infreefall. If a freefall condition is detected, the accelerometerprovides a suitable signal to the processor 500, and the processorinitiates automatic retraction of the nib via the motor 602.

Referring to FIG. 34, there is shown schematically a second automaticretraction mechanism 610 where the motor 602 is coupled directly to aforce sensor assembly 612 to effect extension/retraction of thecartridge 402. The force sensor assembly 612 comprises a housingcontaining a boot 614, which is preloaded against a force sensor. Thecartridge 402 is retained by the boot 614 so that actuated movement ofthe force sensor assembly 612 by the motor 602 causesretraction/extension of the cartridge 402. An advantage of coupling themotor 601 directly to the force sensor assembly 612 is that no ratchetmechanism is required.

In either of the first or second automatic retraction mechanisms, themotor coupling may be geared to increase torque. Forward and reversestop positions may be detected by standard motor load detectionmechanisms.

Referring to FIG. 35, there is shown schematically a third automaticretraction mechanism 620, where the pen retains a manualextension/retraction mechanism whilst also incorporating an automaticretraction mechanism. The cartridge 402 is retained by a boot 614, whichis part of the force sensor assembly 612. The force sensor assembly 612may be pushed forwards, effecting extension of the nib, by amanually-operated extension plunger 622. The force sensor assembly 612is held in place in this extended position by a sprung catch 624, whichabuts with an abutment surface or lip 626 on the force sensor assembly.The catch 624 provides a reaction to the biasing force of a load spring628 acting on the force sensor assembly. The extension plunger 622 isdisengaged from the force sensor assembly 612 after extension by aspring 630.

Automatic retraction is effected by an actuator that lifts the forcesensor assembly 612 off the catch 624, causing the force sensor assemblyand hence the nib to spring backwards into the retracted position. Theactuator may be implemented as a motorized cam 632 as shown in FIG. 35.The processor 500, which may receive input signals from an off switch634 and/or the accelerometer 606, controls the operation of themotorized cam 632, thereby controlling automatic nib retraction.

3.5.5 Main Assembly 480

There are three PCBs in the pen 400: the main PCB 408, the optics PCB431, and a gesture flex PCB 481 (which also locates the indicator LEDs420).

In the present discussion, the main assembly 480 is taken to include thegesture flex PCB 481, a cradle contacts lead-frame insert 482, IR LEDs416 and the battery 410, all of which are connected to the main PCB 408.Of course, the optics PCB 431 and force sensor 442 are also connected tothe main PCB 408, but these components are discussed above separately.

FIG. 36 is an interconnect diagram showing the main connections for thevarious components connected to the main PCB 408.

3.5.5.1 Main PCB 408

Referring to FIGS. 37A and 37B, the main PCB 408 is a 6-layer FR4 1.0 mmthick PCB with minimum trace width and separation of 100 microns. Viaspecification is 0.15 mm hole size in a 0.30 mm pad. The main PCB 408 isa rectangular board with dimensions 108 mm×14 mm.

The main processors (Jupiter 505 and Atmel ARM7 500) are soldered to themain PCB 408, together with a real-time clock (RTC), CSR Bluetooth chipand shielding can.

There are five connectors 483 soldered onto the main PCB 408. A firstconnector 483A is for the optics PCB 431, a second connector 483B is forthe cradle contact lead-frame insert 482 and IR LEDs 416, a thirdconnector 483C is for the gesture flex PCB 481, a fourth connector 483Dis for the battery 410, and fifth connector 483E is for the force sensorflex 443. Cable harnesses to the cradle contact lead-frame insert 482,the battery 410 and the infrared LEDs 416 are the only wired cablesinside the pen 400.

Also soldered onto the main PCB 408 is a reset switch (not shown). Thisis accessible at the underside of the pen through an opening largeenough to admit a paperclip.

3.5.5.2 Cradle Contacts Leadframe Insert 482

Referring to FIG. 38, the cradle contacts leadframe insert 482 forms aset of 4 contacts corresponding to USB signals that are exposed on theunderside of the pen 400, and provides a set of flying leads 484 forconnection to the main PCB 408. The 8-way connector 483B integrates boththe flying leads 484 for the cradle contacts leadframe insert 482, andflying leads 484 for the infrared LEDs 416 situated in the nosecone 409.

The leadframe insert 482 comprises the external contacts 424, which aregold plated to reduce the possibility of corrosion. The contacts 424 arepositioned in the pen's housing 404 such that reliable contact is madebetween the contacts and a corresponding set of contacts in the Netpagepen cradle 426 when the Netpage pen 400 is inserted into the cradle.

3.5.5.3 Gesture Flex PCB 481

Referring to FIG. 39, the gesture flex PCB 481 is a flex PCB onto whichare soldered a Gesture button 484, and one or more user feedback LEDpackages 420 (e.g. one for each of the battery status, online status,and capture blocked indicators).

The gesture flex PCB 481 is formed so that it wraps around the inside ofthe pen housing 404. The bulk of the gesture flex PCB 481 is situated onthe top surface of the Netpage pen 400 within the housing 404, allowingthe indicator LEDs 420 to be positioned under their correspondingapertures, and allowing mating of the Gesture button 485 with acorresponding button molding in the case.

The flex 481 is a 2-layer polyimide PCB, nominally 75 microns thick. ThePCB is specified as flex on install only, as it is not required to moveafter assembly of the pen 400. Stiffener is placed under the Gesturebutton 485, and also at the connector (to the main PCB) to make it thecorrect thickness for the flex connector 483C used on the main PCB 408.

3.6 Optical Design

The pen incorporates a fixed-focus narrowband infrared imaging system.It utilises a camera with a short exposure time, small aperture, andbright synchronised illumination to capture sharp images unaffected bydefocus blur or motion blur.

TABLE 9 Optical Specifications Magnification ^(~)0.248 Focal length oflens 6.069 mm Total track length 41.0 mm Aperture diameter 0.7 mm Depthof field ^(~)/5.0 mm^(a) Exposure time 100 us Wavelength 810 nm^(b)Image sensor size 256 × 256 pixels Pixel size 8 um Pitch range^(c)^(~)22.5 to 45 deg Roll range ^(~)45 to 45 deg Yaw range 0 to 360 degMinimum sampling rate 2.0 pixels per macrodot Maximum pen velocity 0.5m/s ^(a)Allowing 63.5 um blur radius ^(b)Illumination and filter^(c)Pitch, roll and yaw are relative to the axis of the pen.

3.6.1 Pen Optics Design Overview

Cross sections showing the pen optics are provided in FIGS. 40A and 40B.An image of the Netpage tags 4 printed on the surface 1 adjacent to thenib 406 is focused by a lens 436 onto the active region of the imagesensor 432. The small aperture 439 ensures the available depth of fieldaccommodates the required pitch and roll ranges of the pen.

A pair of LEDs 416 brightly illuminate the surface within the field ofview. The spectral emission peak of the LEDs 416 is matched to thespectral absorption peak of the infrared ink used to print Netpage tags4 so as to maximise contrast in captured images of tags. The brightnessof the LEDs 416 is matched to the small aperture size and short exposuretime required to minimise defocus and motion blur.

A longpass filter window 417 suppresses the response of the image sensor432 to any colored graphics or text spatially coincident with imagedtags 4 and any ambient illumination below the cut-off wavelength of thefilter. The transmission of the filter 417 is matched to the spectralabsorption peak of the infrared ink in order to maximise contrast incaptured images of tags 4. The filter 417 also acts as a robust physicalwindow, preventing contaminants from entering the optical assembly 412.

3.6.2 Imaging System

A ray trace of Netpage pen's optic path is shown in FIG. 41.

The image sensor 432 is a CMOS image sensor with an active region of 256pixels squared. Each pixel is 8 um squared, with a fill factor of 50%.

The 6.069 mm focal length lens 436 is used to transfer the image fromthe object plane (paper 1) to the image plane (image sensor 432) withthe correct sampling frequency to successfully decode all images overthe specified pitch, roll and yaw ranges. The lens 436 is biconvex, withthe most curved surface being aspheric and facing the image sensor 432.As discussed in Section 2.10, the minimum imaging field of view requiredto guarantee acquisition of an entire tag 4 has a diameter of 46.7sallowing for arbitrary alignment between the surface coding and thefield of view. Given a macrodot spacing, s, of 127 um, this gives arequired field of view of 5.93 mm.

The required paraxial magnification of the optical system is defined bythe minimum spatial sampling frequency of 2.0 pixels per macrodot forthe fully specified tilt range of the pen, for the image sensor of 8 umpixels. Thus, the imaging system employs a paraxial magnification of−0.248, the ratio of the diameter of the inverted image (1.47 mm) at theimage sensor to the diameter of the field of view (5.93 mm) at theobject plane, on an image sensor of minimum 224×224 pixels. The imagesensor 432 however is 256×256 pixels, in order to accommodatemanufacturing tolerances. This allows up to ±256 um (32 pixels in eachdirection in the plane of the image sensor) of misalignment between theoptical axis and the image sensor axis without losing any of theinformation in the field of view.

The lens 436 is made from Poly-methyl-methacrylate (PMMA), typicallyused for injection moulded optical components. PMMA is scratchresistant, and has a refractive index of 1.49, with 90% transmission at810 nm. The transmission is increased to 98% by an anti-reflectioncoating applied to both optical surfaces. This also removes surfacereflections which lead to stray light degradation of the final imagecontrast. The lens 436 is biconvex to assist moulding precision andfeatures a mounting surface to precisely mate the lens with the opticalbarrel assembly.

A 0.7 mm diameter aperture 439 is used to provide the depth of fieldrequirements of the design. The specified tilt range of the pen is−22.5° to +45.0° pitch, with a roll range of −45.0° to +45.0°. Tiltingthe pen through its specified range moves the tilted object plane up to5.0 mm away from the focal plane. The specified aperture thus provides acorresponding depth of field of ±5.0 mm, with an acceptable blur radiusat the image sensor of 15.7 um. To accommodate the asymmetric pitchrange, the focal plane of the optics is placed 1.8 mm closer to the penthan the paper. This more nearly centralizes the optimum focus withinthe required depth of field.

The optical axis is parallel to the nib axis. With the nib axisperpendicular to the paper, the distance between the edge of the fieldof view closest to the nib axis and the nib axis itself is 2.035 mm.

The longpass filter 417 is made of CR-39, a lightweight thermosetplastic heavily resistant to abrasion and chemicals such as acetone.Because of these properties, the filter 417 also serves as a window. Thefilter is 1.5 mm thick, with a refractive index of 1.50. Like the lens,it has a nominal transmission of 90% which is increased to 98% with theapplication of anti-reflection coatings to both optical faces. Eachfilter 417 may be easily cut from a large sheet using a CO₂ lasercutter.

3.6.3 Illumination System

The tagged surface 1 is illuminated by a pair of 3 mm diameter LEDs 416.The LEDs 416 emit 810 nm radiation with a divergence half intensity,half angle of ±15 degrees in a 35 nm spectral band (FWHM), each with apower of approximately 45 mW per steradian.

3.7 Software Design 3.7.1 Netpage Pen Software Overview

The Netpage pen software comprises that software running onmicroprocessors in the Netpage pen 400 and pen cradle 426.

The Netpage pen 400 contains a number of microprocessors, as detailed inthe Section 3.8. The Netpage pen 400 software includes software runningon the Atmel ARM7 CPU (hereafter CPU), and software running in the VM onthe CSR BlueCore Bluetooth module (hereafter pen BlueCore). Both ofthese processors has an associated flash memory which stores theprocessor specific software, together with settings and other persistentdata. The pen BlueCore also runs firmware supplied by the modulemanufacturer, and this firmware is not considered a part of the Netpagepen software.

The Netpage pen cradle 426 contains a CSR BlueCore Bluetooth module(hereafter cradle BlueCore). The Netpage pen software also includessoftware running in the VM on the cradle BlueCore.

As the Netpage pen 400 traverses a Netpage tagged surface 1, a stream ofcorrelated position and force samples are produced. This stream isreferred to as Digital Ink.

The Netpage pen software is responsible for Digital Ink capture, tagimage processing and decoding (in conjunction with the Jupiterco-processor 505 and Himalia image sensor 432), force sample processing(in conjunction with the Jupiter co-processor) storage and offloadmanagement, host communications and encryption, user feedback, softwareupgrade, battery charging, and Bluetooth management. It includes areal-time operating system (RTOS) and relevant hardware drivers. Inaddition, it provides a manufacturing and maintenance mode forcalibration, configuration or detailed (non-field) fault diagnosis.

The cradle BlueCore VM software is responsible for controlling theNetpage pen cradle's LEDs appropriately, handling Bluetooth pairingoperations, and communicating with the host PC via USB. A more detaileddescription of the software modules is given below.

The Netpage pen software is field upgradable, with the exception of theinitial boot loader. When then Netpage pen 400 is in placed intomaintenance mode by a user, a software upgrade may be performed overBluetooth. After being received and validated, the new software image isused on the next boot of the Netpage pen.

3.7.2 Netpage System Overview

The Netpage pen software is designed to operate in conjunction with alarger software system. A Netpage pen cradle 426 (or 3rd party Bluetoothadapter) receives Digital Ink from the Netpage pen 400 via a Bluetoothwireless connection and relays this to the Netpage server 10 by way ofan attached desktop or laptop computer (see, for example, FIG. 4). It ispossible that the Netpage server 10 will be co-resident in the attacheddesktop or laptop computer.

The Netpage server 10 persists Digital Ink permanently, and providesboth application services for Digital Ink based applications (such ashandwriting recognition and form completion), and database functionalityfor persisted Digital Ink (such as search, retrieval and reprinting).

3.7.3 Internal Design

The Netpage pen software is divided into a number of major modules:

-   -   Image Processing    -   Digital Ink storage and offload management    -   Host Communications    -   User Feedback    -   Power Management    -   Real Time Operating System    -   Hardware Drivers    -   Manufacturing and Maintenance mode (including Software Upgrade)    -   Force Sensor processing    -   Netpage pen BlueCore VM software    -   Netpage Smart Cradle BlueCore VM software        The remainder of this section provides a brief overview of these        major software modules.

3.7.3.1 Image Processing

The position information in the Digital Ink stream is produced when theNetpage pen 400 traverses a surface 1 encoded with the Netpage surfaceencoding scheme. Images of the surface are captured by the Jupiterco-processor and Himalia image sensor 432 which are analysed to extractthe required position information.

The Image Processing module is responsible for analysing images capturedby Jupiter, identifying and decoding tags 4, estimating the pose of thepen 400, and combining this information to obtain position samples.

3.7.3.2 Digital Ink Storage and Offload Management

Any Digital Ink which corresponds to physical marking of an encodedsurface 1 must be reliably and transactionally recorded by the Netpagesystem to allow for accurate reproduction of the surface, includingcaptured Digital Ink, when it is subsequently displayed or reprinted.Ensuring that Digital Ink is reliably recorded is the responsibility ofthe Digital Ink storage and offload management software.

Digital Ink is persisted into non-volatile flash memory on the Netpagepen, and arranged for offload to a Netpage server. This offload processis transactional—the Netpage pen software maintains its record ofDigital Ink until it can guarantee that the Digital Ink has beenreceived and persisted by the Netpage server 10.

Digital Ink may also be streamed to the Netpage server 10 as it iscaptured in order to reduce latency—in this case, the Digital Ink mayalso be temporarily stored within the Netpage pen's internal storageunless it is pre-emptively received and persisted by the Netpage server.

3.7.3.3 Host Communications

The Netpage pen software communicates with the Netpage server 10 by wayof wireless Bluetooth communication. Bluetooth connectivity is providedby the pen BlueCore.

The Communications module of the software is responsible for reliablytransmitting Digital Ink from the Digital Ink storage and offloadmanagement module to the Netpage server 10. It also provides managementfunctionality such as maintaining a record of the Netpage server withwhich the Netpage pen 400 is currently associated, and authenticatingand encrypting communications appropriately. Bluetooth communication isonly performed with the paired Bluetooth device, and uses Bluetoothencryption and authentication facilities to secure these communications.

3.7.3.4 User Feedback

The Netpage pen provides three user feedback indicators that comprise anumber of LEDs 420. The user feedback software module is responsible forconverting signals from other software modules into user feedback usingthe provided mechanisms

3.7.3.5 Power Management

The Netpage pen has a limited power budget, and its design allows fordynamic power saving in a number of ways. For example, the CPU candisable peripherals when they are not in use to save power, and the penBlueCore can be placed into a deep sleep mode. The CPU itself can bepowered down when the pen is not performing higher functions. Indeed,the only always-on components are the ancillary wakeup conditioningcircuits, and pen BlueCore, which can power on the CPU in response toexternal stimuli.

The Power Management module is responsible for analysing the current penstate and optimizing the power usage by switching off un-neededperipherals and other components as required. That is, this moduleintelligently manages the facilities offered by the Netpage pen hardwareto provide optimal power usage given the required Netpage penfunctionality.

3.7.3.6 Software Upgrade

The Netpage pen software is field upgradable, obtaining new softwareimages via its Bluetooth connection. The Software Upgrade module isresponsible for managing the download of complete images via theCommunications module, validating these images against includedchecksums, and arranging for the Netpage pen to boot from a revisedimage when it has been validated.

The Software Upgrade process is only permitted when the Netpage pen isplaced into maintenance mode by a user. Any outstanding Digital Ink maybe offloaded before any software upgrade is initiated, or alternatively,if desired, the Digital Ink may be erased. This simplifies management ofthe internal Digital Ink formats, allowing them to be upgraded asnecessary in new software loads. An existing pairing and associationwith a Netpage server 10 is expected to survive a software upgrade,although under some circumstances it may be necessary to repeat apairing operation.

It should also be noted that small parts of the Netpage pen software,such as basic boot logic, are not field upgradable. These parts of thesoftware are minimal and tightly controlled.

3.7.3.7 Real Time Operating System

The Netpage pen software includes a Real Time Operating System (RTOS)for efficient management of CPU resources. This allows optimal handlingof concurrent Digital Ink capture, persistence, and offload despite thelatencies involved in image capture, flash manipulation, andcommunication resources.

3.7.3.8 Hardware Drivers

The Netpage pen software includes hardware drivers for all peripherals(both internal to the CPU and external to it) required for operation ofthe Netpage pen. This includes USRT, UART and LSS drivers for externalbus communication, as well as higher level drivers for managing theJupiter co-processor and Himalia image sensor 432, the pen BlueCore, andother internal systems.

3.7.3.9 Manufacturing and Maintenance Mode

The Netpage pen may be put into a special manufacturing and maintenancemode for factory calibration or detailed non-field failure analysis. Themaintenance mode is also used to provide in-the-field firmware upgradefunctionality with the assistance of a desktop computer based userinterface.

3.7.3.10 Force Sensor Processing Software

The force sensor 442 in the Netpage pen 400 interfaces to the Jupiterco-processor 505, the output of which is a stream of force samplesproduced at the nominal force sampling rate. The force sensor processingsoftware is tasked with filtering and resampling the force data obtainedfrom the Jupiter co-processor to produce a stream of force samples to beincluded into the Digital Ink stream as recorded by the Netpage pen.

3.7.3.11 Pen Cradle BlueCore VM Software

The Netpage pen cradle 426 contains a CSR BlueCore Bluetooth module. Thecradle BlueCore runs Netpage pen software in its VM. This software isresponsible for controlling the cradle's user feedback LEDs to indicatepower and online status, and managing the USB communication link betweenthe cradle BlueCore and the host PC, and managing Bluetooth pairingoperations. The BlueCore provides a completely embedded stack model forthe Bluetooth RFCOMM stack; hence, the host PC only requires arelatively lightweight USB driver.

3.8 Electronics Design

FIG. 42 is a block diagram of the pen electronics. The electronicsdesign for the pen is based around five main sections. These are:

-   -   The main ARM7 microprocessor 500.    -   The Jupiter co-processor 505 and Himalia image sensor 432.    -   The Bluetooth communications module 510.    -   The real time clock 515.    -   Power supply regulation 520.    -   The force sensor subsystem 442.

3.8.1 ARM7 Microprocessor

The pen uses an Atmel AT91FR40162S microprocessor 500 (see Atmel, AT91ARM Thumb Microcontrollers—AT91FR40162 Preliminary,http://www.keil.com/dd/docs/datashts/atmel/at91fr40162.pdf) running at75 MHz. The AT91FR40162S incorporates an ARM7 microprocessor, 256 kBytesof on-chip single wait state

SRAM and 2 MBytes of external flash memory in a stack chip package.

This microprocessor 500 forms the core of the pen. Its duties include:

-   -   Configuration of the Jupiter co-processor 505 and Himalia image        sensor 432.    -   Decoding images of surfaces encoded with the Netpage surface        encoding with assistance from the image processing features of        the Jupiter co-processor 505.    -   Post-processing of force sensor samples received from the        Jupiter co-processor 505.    -   Compression, encryption and transmission of Digital Ink to a        Netpage server 10 by way of the Bluetooth communications module        510.

The ARM7 microprocessor 500 communicates with the Jupiter co-processor505 using a Universal Synchronous Receiver Transmitter (USRT) with a37.5 MHz clock, and communicates with the Bluetooth module 510 using aUART running at 1.5 Mbps. Communications with the Himalia image sensor432 and RTC 515 use a Low Speed Serial bus (LSS).

The ARM7 microprocessor 500 is provided with bootloader software and isinitially responsive to downloading firmware over the UART interface towhich the Bluetooth module 510 is connected. Firmware programming of theARM7 is performed by way of the Bluetooth module after the Bluetoothmodule has been programmed.

3.8.2 Jupiter Co-Processor 505

The Jupiter co-processor 505 provides two main functions within theNetpage pen's architecture—namely image buffering and processing, andforce sensor interfacing and sample buffering.

Jupiter controls the illumination of a surface by determining thestrobing of two infrared LEDs 416 during the integration time of theHimalia image sensor 432. Once the integration time has elapsed, theimage sensor 432 provides parallel readout of the sensor array by way ofthe image sensor interface (ISI)— as pixels are received, they arefiltered by Jupiter and stored within an internal framebuffer for laterprocessing operations and retrieval by the ARM7 microprocessor 500.

The force sensor 442 connected to Jupiter is pre-filtered and sampled byJupiter such that synchronisation points exist with respect to anyconcurrent image capture activities. The force sensor samples arewritten to an internal FIFO for readout and further processing by theARM7 microprocessor 500.

3.8.3 Bluetooth Communications Module 510

The pen uses a CSR BlueCore5-Multimedia-Flash device as the Bluetoothcontroller 510. It contains an internal 8 Mbit flash memory device tohold its program code and associated VM applications. The BlueCore5meets the Bluetooth v2.0 specification and provides Enhanced Data Rate(EDR) Bluetooth communications at a signalling rate of up to 3 Mbps.

A 2.45 GHz meander-line antenna is incorporated into the Netpage pen 400for the Bluetooth communications.

The BlueCore5 provides a USB device peripheral and the requireddifferential signaling pads. This peripheral is used to allowenumeration of the Netpage pen as a USB device so that charge currentmay be negotiated when connected to a variant of the cradle 426 (i.e. acradle that does not incorporate a Bluetooth communications facility).

The Netpage pen's reset function is controlled by the BlueCore5, whichtakes the user accessible reset switch as an input.

The BlueCore5 incorporates a battery charger to which the lithiumpolymer battery 410 is connected. The battery charger operates in alargely unattended mode, and provides trickle charging, constant currentand constant voltage modes. Charge current is sourced from the cradleconnector. The BlueCore5 is also able to provide a measure of thecurrent battery voltage which is used to both provide a user indicationof remaining useable battery life, and to also prevent use when thebattery is significantly discharged.

The BlueCore5 requires firmware programming with a self-containedBluetooth stack and VM applications that form part of the Netpage penoperating environment. This is performed as part of the manufacturingprocess over an SPI interface that is exposed as pads on the PCB—oncethe BlueCore5 has been programmed, the ARM7 microprocessor 500 issubsequently programmed using same SPI interface and the UARTinterconnect.

3.8.4 Real Time Clock (RTC) 515

The Netpage pen uses an Intersil ISL1208 real time clock (RTC) 515 thatruns off the battery 410 in order to provide the Netpage pen 400 with alocal timesource for timestamping Digital Ink.

The real time clock requires a 32.768 kHz crystal as a timing source.

3.8.5 Power Supply Regulation

The Netpage pen derives all internal power supplies from the internallithium polymer battery 410.

The power supply regulation components generate the following separatevoltages:

-   -   3.1V from an LDO for the ARM7 and BlueCore5 IO voltage.    -   3.1V from an LDO for the Jupiter and Himalia IO, and Himalia        pixel reset voltages.    -   6.2V from a charge pump for the infrared LED drive voltage.

Power supplies for the Bluetooth module, and the 1.8V core voltagesupplies for the Jupiter co-processor, and the ARM7 are provided asoutputs from the Bluetooth module's internal LDOs.

At power up or reset of the Bluetooth module, the ARM7 and BlueCore5 IOvoltage, and 1.8V core voltage are available. The Bluetooth module 510controls enable and disable of the ARM7's clock in accordance with theNetpage pen's current operating mode. Jupiter's core and IO supplies arelikewise controlled by the Bluetooth module, and can be turned on or offdepending on the pen's current operating mode.

The indicator LEDs 420 in the Netpage pen 400 are powered from the IOvoltage supply, and are driven either in PWM mode for those connected tothe ARM7 500, or by a current sink for those connected to the Bluetoothmodule 510.

When the Netpage pen's nib is retracted, the Netpage pen is in itslowest power state—the Jupiter co-processor's supplies are turned off,the ARM7's clock is disabled and the Bluetooth module is powered off(which disables the internal LDOs).

When the Netpage pen's nib is extended, or there is an RTC wake-upalarm, the Bluetooth module receives a power on signal, turns on itsinternal LDOs and boots the ARM7 microprocessor by enabling its clockafter the supplies have stabilised.

3.8.6 Force Sensor System

The force sensor subsystem comprises a custom capacitive force sensor442, and front end filtering and sampling.

The custom capacitive force sensor 442 is sampled at high frequency(typically around 1 MHz), and decimated to a rate of around 1.5 kHz bythe Jupiter co-processor 505. Subsequent processing by the ARM7microprocessor 500 is required to further decimate and low-pass filterthe resulting 10-bit force samples for inclusion in the Digital Inkstream.

The Jupiter co-processor 505 provides a single point factory calibrationof the force sensor 442, and includes a configurable thresholds forperforming pen down and pen up detection based on force sensor input. Atemperature sensor is also present in the Jupiter co-processor 505,allowing for compensation of temperature dependence in the force sensingchain (the temperature sensor is also used for management of batterycharging).

The present invention has been described with reference to a preferredembodiment and number of specific alternative embodiments. However, itwill be appreciated by those skilled in the relevant fields that anumber of other embodiments, differing from those specificallydescribed, will also fall within the scope of the present invention.Accordingly, it will be understood that the invention is not intended tobe limited to the specific embodiments described in the presentspecification, including documents incorporated by cross-reference asappropriate. The scope of the invention is only limited by the attachedclaims.

1. An electronic pen having a retractable nib for interacting with asurface, said pen comprising: a slidably movable cartridge having a nibat a first end for contacting the surface, said cartridge beingconfigurable in an extended position, wherein said nib protrudes from abody of said pen, and a retracted position wherein said nib is retractedin said body; a retraction mechanism comprising: a barrel for receivingan opposite second end of said cartridge, said barrel being adapted forseating said cartridge in either said extended position or saidretracted position, whereby a longitudinal axial force from saidcartridge is transmitted to said barrel in said extended position; and abiasing mechanism for biasing said cartridge towards said extended orretracted positions; an actuator coupled to said retraction mechanism,said actuator enabling a user to move said cartridge between saidextended and retracted positions; a force sensor coupled to said barrel,said force sensor providing a reaction to said axial force transmittedto said barrel from said cartridge; and a processor configured forgenerating force data indicative of a force detected by said forcesensor.
 2. The electronic pen of claim 1, further comprising an imagesensor for imaging at least some of a coded data pattern disposed onsaid surface, wherein said processor is configured for generatinginteraction data using one or more imaged portions of the coded datapattern.
 3. The electronic pen of claim 2, wherein said processor isconfigured to determine, using said force data, either one of: apen-down state when said nib is in contact with said surface; and apen-up state when said nib is not in contact with said surface.
 4. Theelectronic pen of claim 3, wherein said processor is configured toindicate a pen-down state only if said force exceeds a predeterminedthreshold force.
 5. The electronic pen of claim 3, wherein saidprocessor is configured to generate said interaction data only when apen-down state has been determined
 6. The electronic pen of claim 3,wherein said image sensor is configured to sense said coded data onlywhen a pen-down state has been determined.
 7. The electronic pen ofclaim 1, wherein said actuator comprises a manually-operable buttoncoupled to said cartridge, wherein depression and release of said buttonswitches said cartridge between said retracted and extended positions.8. The electronic pen of claim 7, wherein said pen comprises a sensingarrangement for sensing a configuration of said pen, said sensingarrangement comprising: a button sensor coupled to said button, saidbutton sensor sensing actuation of said button; and a cartridge sensorcooperating with said cartridge, said cartridge sensor sensing whethersaid cartridge is extended or retracted, wherein said processor isadapted to configure a state of the pen in response to one or more inputsignals from the sensing arrangement.
 9. The electronic pen of claim 8,wherein said button sensor comprises an electronic switch, said switchbeing mechanically actuable by part of said button.
 10. The electronicpen of claim 1, wherein said barrel comprises a longitudinally slidableratchet for retaining the second end of said cartridge, said ratchetcomprising a plurality of radial ratchet-teeth for longitudinal slidingengagement with complementary longitudinal grooves defined in an innersurface of said barrel.
 11. The electronic pen of claim 1, wherein saidcartridge is replaceable and said ratchet is configured for releasablyretaining said second end of said cartridge.
 12. The electronic pen ofclaim 10, wherein each longitudinal groove has a cammed lip configuredsuch that abutting engagement of a first cammed end-surface of saidratchet-teeth onto said lip causes rotation of said ratchet.
 13. Theelectronic pen of claim 1, wherein said barrel comprises a firstabutment surface corresponding to said extended position and a secondabutment surface corresponding to said retracted position, said ratchetbeing rotatable and slidable towards said first or second abutmentsurface.
 14. The electronic pen of claim 13, wherein said button iscoupled to said ratchet via a plunger, said plunger comprising aplurality of radial plunger-teeth slidingly engaged in said longitudinalgrooves, each plunger-tooth having a second cammed end-surface forabutting engagement with said first cammed end-surfaces of saidratchet-teeth, said first and second cammed end surfaces beingconfigured to impart rotation to said ratchet.
 15. The electronic pen ofclaim 14, wherein said button is biased towards a decoupled position inwhich said button is decoupled from said plunger.
 16. The electronic penof claim 1, wherein said barrel comprises an integral pin engaged withsaid force sensor, wherein a longitudinal axial force transmittedthrough said cartridge is detected by said force sensor via said pin.17. The electronic pen of claim 16, wherein said force sensor is acapacitive force sensor, said pin being engaged with a resilient movingplate of said capacitive force sensor.
 18. The electronic pen of claim16 comprising a preload bias assembly for biasing said barrel andthereby said pin towards engagement with said force sensor.
 19. Theelectronic pen of claim 18 further comprising a housing for said barrel,wherein said preload bias assembly comprises a preload spring engagedbetween said housing and said barrel.
 20. The electronic pen of claim 1,wherein said cartridge is selected from any one of: an ink cartridgecomprising an ink reservoir for supplying ink to said nib, wherein saidnib is a marking nib; and a stylus cartridge wherein said nib is anon-marking nib.